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Abstract 


A  survey  can  characterize  the  knowledge,  attitudes,  and  behaviors  of  a  large  group  of  people 
through  the  study  of  a  subset  of  them.  However,  to  protect  the  validity  of  conclusions  drawn 
from  a  survey,  certain  procedures  must  be  followed  throughout  the  process  of  designing, 
developing,  and  distributing  the  survey  questionnaire. 

Surveys  are  used  extensively  by  software  and  systems  engineering  organizations  to  provide 
insight  into  complex  issues,  assist  with  problem  solving,  and  support  effective  decision 
making. 

This  document  presents  a  seven-stage,  end-to-end  process  for  conducting  a  survey. 
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v 


Document  Overview 


Introduction 


Objective  of  this 
document 


Intended 

audiences 


Document 

format 


Survey-based  research  can  be  used  to  characterize  the  knowledge, 
attitudes,  and  behaviors  of  a  large  group  of  people  through  the  study  of 
a  subset  of  them.  For  this  reason,  surveys  are  used  extensively  by 
software  and  systems  engineering  organizations  to  provide  insight  into 
complex  issues,  assist  with  problem-solving,  and  support  effective 
decision  making. 


A  survey  is  the  collection  of  data  using  a  standardized  questionnaire. 

In  this  document,  the  survey  is  described  as  a  seven-stage  process. 

The  objective  of  this  document  is  to  show  people  how  to  conduct  a 
survey.  An  end-to-end  process  is  outlined  that  describes  each  stage.  By 
following  the  process,  you  will  end  up  with  a  successful  survey. 


The  primary  audience  for  this  document  is  individuals  who  are,  or  will 
be,  responsible  for  designing  and  developing  a  survey.  But,  as 
discussed  in  upcoming  sections,  an  effective  survey  is  typically  the 
result  of  a  team  effort.  All  survey  team  members  would  benefit  from 
understanding  the  survey  process  approach  described  in  this  document. 

The  secondary  audience  for  this  document  is  individuals  who  read  and 
use  survey  results.  Knowing  the  details  of  the  survey  process  can  help 
them  judge  whether  the  survey  method  used  to  produce  the  results  was 
competent. 


This  document  was  designed  to  support  a  broad  audience  of  readers 
with  varying  information  needs.  Each  page  is  labeled  at  the  top  to 
show  readers  where  they  are  in  the  document.  Information  is  chunked 
and  labeled  to  support  scanning,  while  also  providing  accessible  detail 
to  those  who  have  a  more  comprehensive  need  for  information. 

A  number  of  sections  contain  overviews  that  provide  information 
relevant  for  the  upcoming  subsections.  The  overviews  also  act  as 
information  organizers  to  provide  readers  with  advance  notice  of  what 
to  expect  in  those  section. 
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Document  Overview  (continued) 


Document 

organization 


The  document  is  organized  into  the  following  main  sections: 

•  What  is  Survey  Research? 

•  The  Survey  Research  Process 

•  After  the  Survey 

The  “What  is  Survey  Research?”  section  provides  an  introduction  for 
those  who  might  be  wondering  if  a  survey  is  the  best  way  to  conduct 
their  research.  The  strengths  and  limitations  of  surveys  are  discussed 
in  that  section. 

The  core  substance  of  this  document  is  contained  in  the  “Survey 
Research  Process”  section.  The  process  is  described  as  a  seven-stage 
approach  and  is  discussed  in  seven  subsections. 

The  “After  the  Survey”  section  briefly  discusses  typical  post-survey 
activities. 


In  this 
document 


Section  title 

See  Page 

What  is  Survey  Research? 

3 

The  Survey  Research  Process 

7 

After  The  Survey 

107 

Appendices 

109 

References 

127 
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What  is  Survey  Research? 


Introduction 


Approaches  to 
data  gathering 


What  is  a  survey? 


What 

distinguishes 
a  survey 
from  other 
approaches? 


Using  surveys  is  one  way  to  conduct  research.  But  before  you  decide 
to  invest  your  time  and  resources  in  the  survey  approach,  consider 
whether  a  survey  is  the  best  way  to  find  out  what  you  want  to  know. 

This  section  describes  survey  research  and  why  you  might  want  to  use 
it  for  your  research  project. 


Survey  research  is  but  one  approach  for  collecting  information  to  gain 
insight  into  people  or  problems  under  study.  Other  approaches  include 

•  documentary  research 

•  laboratory  experiments 

•  action  research 

•  case  studies 

•  field  experiments 

•  field  work  (e.g.,  participant  observation  and  ethnography) 

•  simulation 

•  in-depth  interviews 

There  are  strengths  and  weaknesses  to  each  approach.  For  a 
description  and  comparison  of  these  approaches,  see  Appendix  A  on 
page  111. 


A  survey  is  a  data-gathering  and  analysis  approach  in  which 
respondents  answer  questions  or  respond  to  statements  that  were 
developed  in  advance.  In  this  document,  the  approach  is  described  as  a 
seven-stage  process  that  includes  the  following  steps: 

1.  Identify  the  research  objectives. 

2.  Identify  and  characterize  the  target  audience. 

3.  Design  the  sampling  plan. 

4.  Design  and  write  the  questionnaire. 

5.  Pilot  test  the  questionnaire. 

6.  Distribute  the  questionnaire. 

7.  Analyze  the  results  and  write  a  report. 


Surveys  differ  from  other  kinds  of  data-gathering  approaches  in  an 
important  way.  A  survey,  when  conducted  properly,  allows  you  to 
generalize  about  the  beliefs  and  opinions  of  many  people  by  studying 
a  subset  of  them.  However,  a  survey  can  only  be  used  for 
generalization  when  the  survey  process  follows  strict  procedures,  as 
discussed  in  this  document  beginning  on  page  7. 
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What  is  Survey  Research?  (continued) 


Very  often,  the  terms  “questionnaire”  and  “survey”  are  used  to  mean 
the  same  thing.  However,  there  is  a  subtle  but  important  distinction 
between  the  two. 

A  survey  is  a  process.  This  document  describes  a  seven-stage  process 
for  conducting  a  survey.  The  instrument  of  a  survey  is  the  questionnaire. 

While  surveys  always  make  use  of  a  questionnaire,  it  is  the  survey 
process  itself  that  determines  whether  the  information  obtained 
through  the  questionnaire  is  valid  and  can  be  used  to  generalize  about 
the  population  that  the  sample  is  intended  to  represent. 

Surveys  must  be  implemented  following  strict  guidelines  and 
procedures.  Failure  to  follow  these  guidelines  can  lead  to  misleading 
results  that  can  be  challenged  and  refuted.  Developing  and  distributing 
a  questionnaire  in  a  haphazard  fashion  is  not  the  same  as  using  a  well- 
constructed  questionnaire  within  a  carefully  designed  survey  process 


Types  of  surveys  Surveys  fall  into  two  broad  categories: 

•  self-administered  questionnaires 

•  interviews  (distinguished  from  in-depth  interviews,  defined  on  page 
5) 


Are 

questionnaires 
and  surveys 
the  same? 


Self-administered  When  most  people  think  of  a  survey,  they  think  of  a  self-administered 

questionnaires  questionnaire.  This  survey  is  the  kind  you  might  receive  through  the 

mail  or  complete  on  a  set  of  Internet  Web  pages. 

This  document  focuses  on  developing  self-administered 
questionnaires. 

Interviews  A  survey  interview  is  one  in  which  the  interviewer  asks  questions  from 

a  prepared  questionnaire  and  records  the  information.  Interviews  can 
be  conducted  by  the  researcher  working  directly  with  the  respondent 
face-to-face  or  by  telephone.  An  obvious  disadvantage  of  this  method 
is  that  it  is  both  time-consuming  and  costly  for  the  researcher.  You 
would  likely  need  a  team  of  trained  interviewers  to  conduct  the  survey 
interviews. 

We  will  not  discuss  interviewing  methods  and  the  skills  required  to 
implement  interviews  in  this  document.  Oishi  provides  a  detailed 
discussion  of  interviewing  skills  associated  with  surveys  [Oishi  02]. 
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What  is  Survey  Research?  (continued) 


Comment:  survey  Survey  interviews  are  distinguished  from  in-depth  interviews  by  their 

interviews  vs.  in-  objectives  and  by  the  way  in  which  they  are  conducted. 

depth  interviews 

In  survey  interviews,  the  researcher  works  from  a  questionnaire  and 
the  questions  are  presented  to  each  respondent  in  exactly  the  same 
way.  The  objective  is  to  make  the  interviews  as  repeatable  as  possible 
so  that  the  results  can  be  quantified  and  compared. 

When  conducting  in-depth  interviews,  the  researcher  may  work  from  a 
list  of  topics  and  possible  questions,  but  the  interview  is  free-flowing 
and  controlled  as  much  by  the  respondent  as  it  is  by  the  researcher. 
Unanticipated  issues  might  arise  during  the  course  of  the  interview  and 
might  need  to  be  addressed  by  new  questions  the  researcher  did  not 
plan. 


In-depth 
interviews 
as  a  prelude  to 
a  survey 


Quite  often,  conducting  several  in-depth  interviews  with  subject  matter 
experts  as  a  prelude  to  designing  a  survey  instrument  is  valuable.  This 
is  especially  important  when  the  topic  areas  are  fuzzy  and  not  well- 
understood. 

In-depth  interviews  can  help  the  researcher  define  relevant  survey 
questions  that  would  be  otherwise  overlooked. 


Characteristics  The  following  table1  summarizes  important  characteristics  of  surveys. 

of  surveys  _ 


Characteristic 

Description 

Systematic 

The  survey  follows  a  specific  set  of  rules;  a 
formal  and  orderly  logic  of  operations. 

Impartial 

The  survey  selects  units  of  the  population 
without  prejudice  or  preference. 

Representative 

The  survey  includes  units  that  together  are 
representative  of  the  problem  under  study  and  the 
population  affected  by  it. 

Theory-Based 

The  survey’s  operations  are  guided  by  relevant 
principles  of  human  behavior  and  mathematical 
laws  of  probability  and  statistics. 

Quantitative 

The  survey  assigns  numerical  values  to  non- 
numerical  characteristics  of  human  behavior  in 
ways  that  permit  uniform  interpretation  of  these 
characteristics. 

Replicable 

Other  people  using  the  same  methods  in  the  same 
ways  can  get  essentially  the  same  results. 

1  Information  from  this  table  is  adapted  from  Backstrom  [Backstrom  81,  (pages  3-4)]. 
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What  is  Survey  Research?  (continued) 


Limitations  of 
surveys 


While  surveys  can  provide  significant  advantages  over  other  data- 

collecting  approaches,  there  are  certain  limitations  that  should  be 

understood. 

•  To  generalize  for  a  population,  a  survey  must  follow  strict 
procedures  in  defining  which  participants  are  studied  and  how  they 
are  selected. 

•  Following  the  rules  and  implementing  the  survey  with  the  rigor 
that  is  necessary  can  be  expensive  with  respect  to  cost  and  time. 

•  Survey, data  is  usually  superficial  [Backstrom  81].  It  is  not 
typically  possible  to  go  into  any  detail — that  is,  we  are  not  capable 
of  digging  deeply  into  people’s  psyches  looking  for  fundamental 
explanations  of  their  unique  understandings  or  behaviors. 

•  Surveys  can  be  obtrusive.  People  are  fully  aware  that  they  are  the 
subjects  of  a  study.  They  often  respond  differently  than  they  might 
if  they  were  unaware  of  the  researcher’s  interest  in  them. 

Information  is  self-reported  and  is  not  always  the  undiluted  truth. 
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The  Survey  Research  Process — Overview 


Introduction  This  section  provides  an  overview  of  the  seven-stage  survey  research 

,  process. 


1 .  Identify  the  research 
objectives 


3.  Design  sampling 
plan 


5.  Pilot  test 
questionnaire 


2.  Identify  &  characterize 
target  audience 


4.  Design  &  write 
questionnaire 


6.  Distribute 
questionnaire 


7.  Analyze  results 
and  write  report 


Stage 

Description 

1 

Identify  research 
objectives 

What  do  you  want  the  survey  to  accomplish?  What  information  already 
exists  about  the  problem  you  are  asking  questions  about?  Survey 
research  must  begin  with  a  statement  of  the  problem  and  how  the 
survey  will  answer  questions  about  the  problem. 

2 

Identify  &  characterize 
target  audience 

Who,  specifically,  will  respond  to  the  survey?  What  assumptions  can 
you  make  about  their  knowledge  of  the  questions  you  have  in  mind, 
the  terminology  they  understand,  their  willingness  to  participate  in  the 
survey,  and  so  forth? 

3 

Design  sampling  plan 

How  big  is  the  target  audience  population?  Can  the  target  audience  be 
enumerated?  How  will  you  ensure  that  those  who  respond  to  the 
survey  are  representative  of  the  target  audience? 

4 

Design  &  write 
questionnaire 

The  survey  objectives  and  internal  questions  must  be  translated  into 
carefully-worded  questionnaire  items  crafted  to  facilitate  analysis  and 
interpretation. 

5 

Pilot  test  questionnaire 

The  questionnaire  instrument  must  be  “tested”  with  members  of  the 
target  audience  to  remove  bugs  and  improve  the  instrument. 

6 

Distribute  the 
questionnaire 

The  questionnaire  should  be  distributed  to  selected  members  of  the 
target  audience  as  defined  by  the  sampling  plan. 

7 

Analyze  results  and 
write  report 

The  results  should  be  collected  and  translated  into  appropriate 
graphical  displays  that  facilitate  understanding.  The  charts  can  be 
compiled  into  a  report  and  interpretations,  inferences,  generalizations, 
and  caveats  can  be  made  based  on  evidence  provided  by  the  results. 
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The  Survey  Research  Process — Overview  (continued) 


The  need 
for  a  process 
perspective 


As  stated  by  the  preeminent  statistician,  W.  Edwards  Deming: 

The  information  that  a  statistical  survey  or  experiment  provides 
is  not  right  because  someone  agrees  with  it,  nor  wrong  because 
he  does  not;  the  information  is  useful  or  not  useful  depending 
on  the  information  and  the  questions  that  were  built  in  initially 
in  the  statement  of  the  problem,  and  on  the  skill  that  went  into 
the  preparation,  execution,  and  summary  of  the  survey  or 
experiment.  The  burden  rests  with  the  expert  in  the  subject- 
matter,  and  with  management,  in  the  preparation  of  the 
questionnaire  or  of  the  method  of  test,  to  get  information  that 
they  can  apply  to  the  problem  when  the  results  of  the  survey  or 
experiment  come  in.  [Deming  60] 

Dr.  Deming  makes  several  important  points. 

1.  It  is  the  survey  process  that  determines  whether  the  results  will  be 
useful. 

2.  A  useful  survey  is  the  product  of  several  kinds  of  knowledge 
working  skillfully  together.  You  need  both  of  the  following: 

•  survey  design  knowledge  (with  statistical  knowledge  and  skills) 

•  subject  matter  expertise  in  the  domain  under  investigation 
Survey  design  theory  is  not  a  substitute  for  the  knowledge  of  a 
subject  matter  expert.  Conversely,  knowledge  in  the  subject  matter 
will  not  ensure  that  an  effective  survey  will  be  developed.  You 
must  have  both. 

3.  Those  who  will  use  the  survey  information  to  support  decision¬ 
making  (typically  management)  bear  a  responsibility  in  the 
survey  research  process — to  ensure  that  the  survey  is  asking  the 
questions  that  will  lead  to  problem  understanding  and  effective 
decision-making. 


Researcher  Researcher 
Support 


Subject  Matter  Experts 


Management 
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The  Survey  Research  Process — Overview  (continued) 


Team  approach  is  A  successful  survey  is  typically  a  team-based  effort.  A  team  is  needed 

typically  needed  since,  in  most  cases,  no  single  individual  has  all  the  knowledge  and 

skills  required.  The  following  table  describes  the  key  roles  and 
responsibilities  in  a  survey  research  project. 


Role 

Responsibility 

Researcher 

survey  design  and  statistics  expertise 
owner  and  facilitator  of  the  survey  process; 
author  of  the  survey  plan 

ensures  adherence  to  tenets  of  good  survey  design 
becomes  familiar  enough  with  the  domain  under 
investigation  to  ensure  that  survey  objectives  are 
addressed 

translates  internal  questions  into  questionnaire 
items 

distributes  questionnaire  and  monitors  response 
information 

analyzes  results  and  authors  the  survey  report 

Subject 

matter 

experts 

domain  knowledge  expertise 
define  objectives  of  the  survey  research 
develop  internal  questions2  that  address  the 
objectives 

ensure  survey  questionnaire  items  address 
objectives 

review  all  intermediate  and  final  products  of  the 
survey  process  and  provide  feedback 
participate  in  pre-testing  of  the  survey 
instrument 

Management 

approves  the  survey  objectives 

decides  what  action  will  be  taken  given  various 

survey  result  outcomes 

may  develop  internal  questions  that  address  the 
objectives  ’ 

reviews  all  intermediate  and  final  products  of  the 
survey  process  and  provides  feedback 

Researcher 

support 

(examples) 

computer  programming  support  provided  for 
Web-based  survey 

administrative  support  provided  for  mail-type 

survey 

•  technical  writers  and  graphic  designers  on  your 
team  can  significantly  improve  the  quality  of 
your  survey  products 


2  Internal  questions  are  those  that  the  subject  matter  experts  (or  management)  have  that  are  related 
to  the  survey  objectives.  However,  these  questions  typically  need  to  be  broken  into  more  than  a 
single  question,  simplified,  reworded,  and  translated  into  closed-ended  questions  before  they  can 
be  used  on  the  respondent  questionnaire  as  questionnaire  items. 
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The  Survey  Research  Process — Overview  (continued) 


Need  for  a  plan 


Components  of 
survey  plan 


To  set  the  stage  for  effective  collaboration,  the  researcher  should 
develop  a  plan  that  guides  the  effort  and  establishes  a  shared 
understanding  among  the  participants. 

The  first  version  of  the  plan  should  be  written  at  the  beginning  of  the 
project.  This  helps  to  set  team  members’  expectations  so  that  informed 
commitments  can  be  made.  The  plan  should  be  revised  and  updated  as 
new  information  becomes  known,  such  as  the  details  of  the  sampling 
plan.  The  plan  should  be  revised  when  there  are  changes  and  kept 
under  version  control. 


The  components  of  the  survey  plan  are  listed  below. 


Plan  component 

Description 

Objectives 

objectives  of  the  survey  research 

Team  members 
&  roles 

•  team  member  responsibilities 

•  time  committed  to  project 

Target  audience 

•  specific  target  audience  of  survey 

•  assumptions  about  the  target  audience 

Sampling  approach 

the  approach  for  selecting  the  target  audience 
population  who  will  participate  in  the  survey 

Media 

how  the  survey  will  be  mediated  (e.g.,  mail,  email, 
Web-based,  telephone) 

Activities 

description  of  survey  process  activities  and  how  each 
team  member  will  participate  in  the  activity 

Schedule 

activity  milestones 

Product 

how  the  survey  results  will  be  reported  (e.g.,  in  a 
report  or  presentation,  or  both) 
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The  Survey  Research  Process— Overview  (continued) 


Work  breakdown 
structure 


What  about  a 
quick  and  dirty 
survey? 


Survey  design: 
a  key  concern 


Definition:  validity 


A  schedule  is  a  component  of  a  survey  plan.  An  example  activity 
breakdown  structure  is  provided  in  Appendix  C  on  page  119.  Activity 
durations  are  not  provided  in  Appendix  C  because  they  depend  on 
factors  such  as  the  scope  of  the  survey  research  project. 

Has  anyone  ever  said  to  you,  “Having  some  data  is  better  than 
nothing”?  This  thinking  is  sometimes  used  to  justify  a  rush  to  write  up 
a  quick  and  dirty  questionnaire  to  obtain  some  information.  But  the 
assertion  that  “having  some  data  is  better  than  nothing”  is  fraught  with 
error. 

The  problem  with  a  quick  and  dirty  survey  is  that  you  have  no  notion 
of  how  valid  or  reliable  your  results  are.  You  may  be  lucky  and  have 
developed  a  good  questionnaire — or,  you  may  be  unlucky.  But,  the 
point  is,  you  just  won’t  know  unless  you  follow  a  methodical 
procedure  designed  to  mitigate  threats  to  measurement  validity. 

After  the  survey  results  are  in,  you  are  left  in  a  quandary.  Does  the 
survey  fail  to  show  differences  because  they  do  not  exist?  Or,  is  it 
simply  that  the  survey  was  insensitive  because  it  wasn’t  executed  in  a 
way  that  would  lead  to  coherent  and  analyzable  information?  If  the 
results  do  show  differences,  is  it  because  the  questionnaire  is  biased,  or 
is  it  because  there  are  real  statistical  differences?  The  crux  of  the 
matter  here  is  that  you  cannot  know  unless  you  follow  the  principles  of 
good  survey  design. 

So  the  statement,  “Having  some  data  is  better  than  nothing,”  can 
certainly  be  wrong.  It  is  wrong  because  if  the  data  are  invalid,  then  it 
could  lead  you  to  make  false  conclusions  and  wrong  decisions.  In  such 
a  case,  it  might  have  been  better  to  forego  the  cost  of  conducting  the 
quick  and  dirty  survey — because  it  is  no  more  reliable  than  your  guess 
of  what  the  survey  outcome  would  be. 


To  protect  the  validity  of  conclusions  drawn  from  the  information 
resulting  from  a  survey,  strict  procedures  must  be  used  during  the 
design  process. 


“Validity”  is  defined  as  the  best  available  approximation  to  the  truth  of 
a  given  proposition,  inference,  or  conclusion  [Trochim  01]. 
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The  Survey  Research  Process — Overview  (continued) 


Validity  of  survey  Validity  is  a  critical  concern  of  survey  researchers.  It  is  the  survey 
results  process  itself  that  provides  the  foundation  for  the  conclusions  drawn 

by  the  researcher.  Validity  is  most  related  to  the  research  process. 

There  are  always  threats  to  validity  that  the  researcher  must  avoid  or 
minimize  through  procedure  and  careful  development  of  the 
questionnaire  instrument.  In  the  absence  of  these  important 
precautions  (which  are  described  in  later  sections),  assertions  or 
conclusions  drawn  from  the  survey  data  may  be  challenged. 

Two  important  types  of  validity  in  survey  research  are 

construct  validity  Are  we  measuring  what  we  think  we  are 
measuring? 

external  validity  Can  the  results  be  generalized  to  other 
people,  places,  or  times? 

The  body  of  guidance  in  this  document  is  intended  to  provide  ways  for 
improving  the  measurement  validity  of  your  survey  results.3 


In  this  section 


Section 

See  page 

1 

Identify  the  research  objectives 

13 

2 

Identify  &  characterize  target  audience 

17 

3 

Design  sampling  plan 

21 

4 

Design  &  write  questionnaire 

33 

5 

Pilot  test  questionnaire 

75 

6 

Distribute  the  questionnaire 

79 

t  7 

i 

Analyze  results  and  write  report 

83 

3  In  particular,  information  described  in  substage  4.2.2  beginning  on  page  9  specifically  addresses 
ways  of  mitigating  threats  to  construct  validity.  Stage  3  of  the  survey  process  (beginning  on  page 
9)  describes  procedures  that  must  be  applied  to  support  external  validity. 
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1  Identify  the  Research  Objectives 


1.  Identify  the  research 
objectives 


3.  Design  sampling 
plan 


5.  Pilot  test 
questionnaire 


2.  Identify  &  characterize 
target  audience 


4.  Design  &  write 
questionnaire 


6.  Distribute 
questionnaire 


7.  Analyze  results 
and  write  report 


Introduction  A  natural  tendency  is  to  jump  into  action  and  begin  writing  questions 

for  the  questionnaire  the  moment  the  decision  to  do  a  survey  is  made. 
However,  successful  questionnaires  result  from  planning  based  on  a 
careful  examination  of  the  problem  that  provided  the  motivation  to  do 
the  survey  in  the  first  place. 


What  are  survey  Objectives  are  derived  from  a  careful  understanding  of  the  problem  or 
objectives?  jssue  under  study. 

The  survey  research  objectives  provide  scope  and  guidance  for  internal 
questions  the  team  has  about  the  problem  or  issue. 

In  some  cases,  team  members  have  difficulty  defining  objectives  but 
can  provide  internal  questions  (sometimes  called  research  questions) 
that  are  used  to  inform  the  research  objectives.  This  represents  a 
bottom-up  sort  of  thinking  where  the  objectives  are  defined  as  a  result 
of  first  identifying  the  questions. 

Internal  questions  will  not  appear  in  the  actual  survey  questionnaire. 
Internal  questions  are  seldom  expressed  in  the  way  a  respondent  would 
understand,  so  they  must  be  translated  into  the  target  audience  language. 

Also,  internal  questions  can  be  complex  and  open-ended  and  must  be 
broken  into  multiple  simpler  questionnaire  items  that  are  closed-ended. 
(Although  well-scoped  carefully  worded  open-ended  questions  can 
also  provide  useful  information  when  used  sparingly.) 
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1  Identify  the  Research  Objectives  (continued) 


Why  clear 
objectives 
are  needed 


Content  of 
objectives 


Avoid  too  many 
objectives 


Has  someone 
done  this  before? 


The  research  objectives  provide  the  scope  and  guidance  for  several 
aspects  of  the  survey  project.  They  not  only  determine  who  the 
respondents  should  be,  but  also  what  questions  should  be  asked. 

Do  not  make  the  mistake  of  assuming  that  the  research  objectives  are 
commonly  understood  by  everyone  on  the  team.  The  objectives  must 
be  written  down  to  test  whether  or  not  everyone  is  on  the  same  page. 
Taking  the  time  to  do  this  in  the  beginning  will  help  you  avoid  timely 
rework  later  when  the  questionnaire  instrument  is  being  developed. 

In  some  cases,  identifying  the  research  objectives  is  easy  and 
straightforward.  In  other  cases,  you  might  find  that  there  is  actually  a 
wide  disparity  of  viewpoints  that  must  be  reconciled.  Work  sessions 
might  be  needed  to  negotiate  or  clarify  what  the  survey  is  intended  to 
accomplish. 


Your  objectives  should  state  what  you  intend  to  accomplish  with  your 
survey  as  clearly  as  possible.  If  your  objectives  are  fuzzy  or  ill- 
defined,  you  will  struggle  when  you  try  to  develop  appropriate 
questions  and  will  need  to  do  more  work  on  your  objectives. 

In  addition  to  documenting  what  you  hope  to  accomplish  with  the 
survey,  you  should  also  define  and  document  how  the  information  that 
results  from  the  survey  will  be  used.  This  can  help  you  sharpen  your 
thinking  about  the  objectives. 


One  of  the  most  common  challenges  facing  researchers  is  the  pressure 
to  expand  the  scope  of  the  survey  research.  There  are  always  team 
members  who  will  view  the  survey  project  as  an  opportunity  to  collect 
all  kinds  of  information.  Nonetheless,  there  are  practical 
considerations  that  limit  both  the  type  and  amount  of  information  that 
can  be  collected  using  a  single  survey.  These  considerations  are 
explained  in  more  detail  later  the  document,  but  here  it  suffices  to  say 
that  there  are  natural  constraints  to  the  size  and  complexity  of 
questionnaire  instruments. 


As  your  objectives  take  shape,  an  obvious  question  to  ask  is,  “Have 
these  research  objectives  already  been  addressed  by  the  work  of 
others?”  By  conducting  a  literature  review,  you  might  find  that  others 
have  indeed  addressed  aspects  of  your  problem,  if  only  in  part. 

Take  advantage  of  the  research  of  others  to  change  or  refine  your 
objectives.  At  the  least,  other  research  can  be  used  to  inform  your 
objectives  and  to  provide  ideas  about  how  you  will  develop 
questionnaire  items. 
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1  Identify  the  Research  Objectives  (continued) 


Avoid  overzealous 
objectives 


It  is  unreasonable  to  expect  to  include  questions  about  everything  you 
might  want  to  learn  about  in  a  single  survey.  Your  objectives  need  to 
set  reasonable  expectations  for  what  can  be  accomplished  in  a  single 
survey. 

Typically,  a  survey  cannot  provide  conclusive  answers  to  all  questions 
you  have  about  a  complex  problem  or  issue.  This  is  especially  the  case 
for  an  initial  survey,  sometimes  called  an  “exploratory  survey,”  which 
is  one  component  of  an  overall  research  strategy.  Follow-on 
information-gathering  activities  are  used  to  probe  the  problem  space  in 
order  to  elicit  details  about  specific  aspects  of  the  problem  that  the 
initial  survey  illuminated.  These  activities  might  include  follow-up 
surveys  or  other  types  of  data-collection  methods  including  in-depth 
interviews,  case  studies,  or  literature  review. 

Researchers  refer  to  this  approach  as  triangulation.  Rather  than  rely  on 
a  single  data-gathering  event  to  make  concrete  conclusions, 
triangulation  is  used  to  improve  the  confidence  of  assertions  made  by  a 
single  survey. 


Interviews 


Other 

Research 
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1  Identify  the  Research  Objectives  (continued) 
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2  Identify  &  Characterize  the  Target 
Audience 


Introduction  This  stage  of  the  process  is  a  two-step  activity  that  begins  with 

identifying  the  population  you  are  trying  to  describe.  Once  identified, 
this  population  is  explicitly  characterized  in  ways  that  assist  effective 
survey  development. 

Population  vs.  In  survey  research,  a  “population”  refers  to  all  the  members  of  a 
sample  specific  group.  A  population  can  be  defined  in  terms  of  demography, 

geography,  occupation,  time,  or  some  combination  of  these  factors. 

When  you  identify  the  population  for  your  study,  it  then  becomes  the 
target  audience  for  the  research. 

A  “sample”  is  a  subset  of  the  population.  In  survey  research,  the 
sample  is  studied  and  the  findings  are  generalized  to  the  population. 
However,  this  is  only  possible  when  strict  procedures  (described  in 
stage  3)  are  followed  to  ensure  that  the  sample  is  representative  of  the 
population. 


Selecting  your  Select  your  target  audience  based  on  whose  perspective  you  are 
target  audience  interested  in  obtaining.  That  is,  for  the  research  objectives  you  have 
identified,  who  can  best  provide  you  with  the  information  you  need? 
Answering  this  question  will  indicate  who  your  population  is. 

For  example,  consider  the  four  categories  of  people  that  might  be  part 
of  your  organization: 

•  management 

•  engineering  staff 

•  financial  analysts 

•  clerical  staff 
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2  Identify  &  Characterize  the  Target  Audience  (continued) 


Selecting  your 
target  audience, 
continued 


Analyzing  the 
audience 


Understanding 
the  target 
audience 


Conducting 
the  audience 
analysis  meeting 


For  the  objectives  of  a  particular  survey,  you  might  want  the 
perspectives  of  only  management  and  engineering  staff.  So,  for 
these  objectives,  your  population  and  target  audience  will  include 
management  and  engineering  staff. 

For  a  different  type  of  survey  with  different  objectives,  you  might 
define  the  population  as  all  four  groups,  so  your  target  audience  will 
include  them  all. 

Therefore,  the  target  audience  you  select  depends  on  the  problem  you 
are  trying  to  understand  and  who  can  provide  that  information  to  you. 
For  example,  you  might  be  interested  in  obtaining  only  the  opinions  of 
project  managers  in  companies  that  perform  contracting  services  for 
the  Department  of  Defense  (DoD). 

It  is  important  to  be  very  explicit  about  who  the  intended  respondents  for 
your  survey  are.  This  assumption  has  a  dramatic  influence  on  the  design 
of  the  survey  instrument  and  the  distribution  method  for  the  survey. 


Once  you  have  identified  the  target  audience  for  the  survey,  conduct  an 

audience  analysis  to  characterize  the  intended  respondents. 

An  audience  analysis  is  conducted  for  a  number  of  reasons. 

Understanding  the  audience  will  help  you 

•  choose  the  method  of  surveying  (e.g.,  paper  and  pencil,  survey 
interview,  Web-based  questionnaire)  that  will  most  likely  lead  to 
high  response  rates  and  valid  response  information 

•  develop  questionnaire  items  that  can  be  interpreted  by  the 
respondents 


Developing  questionnaire  items  that  can  be  interpreted  by  the 
respondents  is  particularly  important  and  is  worthy  of  additional 
elaboration.  Questionnaire  items  must  be  written  from  the  perspective 
of  the  respondent,  not  the  perspective  of  the  researcher. 

This  means  you  should  ask  questions  you  believe  the  audience  has 
knowledge  about  and  can  answer.  Also,  it  means  that  special  care 
should  be  given  to  writing  questionnaire  items  using  vocabulary 
familiar  to  the  respondent. 


In  many  cases,  you  might  not  have  insight  into  the  characteristics  of  the 
target  audience.  Even  when  this  is  only  partially  true,  it’s  always  a  good 
idea  to  enlist  the  help  of  subject  matter  experts  who  can  provide  opinions 
about  the  target  audience  to  augment  your  existing  knowledge  base. 

Consider  holding  a  team  brainstorming  meeting  to  help  you 
characterize  the  audience. 
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2  Identify  &  Characterize  the  Target  Audience  (continued) 


What  you  will 
want  to  know 


How  is  this 
information 
used? 


Some  example  questions  you  can  use  to  elicit  characteristics  about  the 
target  audience  for  the  survey  are  listed  below. 

•  How  many  people  are  in  the  population  we  are  studying? 

•  What  are  their  jobs  and  responsibilities? 

•  What  is  the  most  common  education  level? 

•  What  relevant  experience  do  they  possess? 

•  What  technical  abilities  do  they  possess? 

•  What  is  the  age  range  of  the  respondent  population? 

•  Do  we  anticipate  that  they  would  have  difficulty  with  using  a 
questionnaire  that  is: 

-  mailed  to  them? 

-  completed  using  a  computer  via  the  internet? 

-  handed  to  them? 

•  What  can  we  assume  about  their  knowledge  of  the  domain  we  are 
studying  in  the  survey? 

•  How  do  they  speak  about  the  domain  under  study?  Do  they  have  a 
special  language  they  use  to  talk  about  the  problem  space? 

•  Will  they  be  motivated  to  complete  the  questionnaire?  Or  will  they 
be  resistant? 

How  much  of  their  time  can  we  assume  they  will  spend  completing 
the  questionnaire? 


If  an  item  from  the  questionnaire  is  misinterpreted  by  the  respondent, 
the  data  you  obtain  through  the  survey  will  be  erroneous  and 
misleading. 

The  information  derived  from  the  audience  analysis  is  indispensable 
when  you  are  designing  and  writing  the  questionnaire  instrument 
because  it  will  help  you  understand  the  precautions  you  must  take  to 
word  the  questions  in  a  way  that  the  respondent  understands. 
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2  Identify  &  Characterize  the  Target  Audience  (continued) 
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3  Design  the  Sampling  Plan 


1.  Identify  the  research 
objectives 


5.  Pilot  test 
questionnaire 


2.  Identify  &  characterize 
target  audience 


4.  Design  &  write 
questionnaire 


6.  Distribute 
questionnaire 


7.  Analyze  results 
and  write  report 


Introduction  During  this  stage,  the  researcher  should  determine 

•  how  individuals  will  be  selected  to  participate  in  the  survey 

•  the  required  size  of  the  sample 

These  considerations  have  significant  consequences  for  how  the 
results  can  be  generalized  beyond  the  sample  and  for  how  much 
precision  and  confidence  you  can  express  about  the  findings. 


What  kind  of 
sample  will  you 
select? 


During  this  stage,  the  researcher  determines  how  the  questionnaire  will 
be  distributed  to  the  population  of  potential  respondents.  Two  primary 
considerations  that  will  drive  your  decision  about  this  are  listed  below. 

•  Are  the  survey  findings  intended  to  be  generalized  to  a  larger 
population?  If  so,  then  a  probability  sample  will  be  required. 

•  Will  the  survey  findings  be  considered  unique  to  the  individuals 
who  participate  in  the  survey?  If  so,  then  a  non-probability  sample 
will  suffice. 


Census: 
a  special  case 


A  “census”  is  a  sample  that  includes  all  individuals  in  the  population. 
A  census  eliminates  sampling  error  and  provides  information  on  all 
individuals  in  the  population.  In  this  case  the  sample  is  the  population. 


What  are 

probability 

samples? 


In  statistics,  the  term  “probability”  means  “chance.”  For  a  population 
of  individuals  under  study,  what  is  a  person’s  chance  of  being  selected 
for  the  survey? 

A  probability  sample  is  designed  to  ensure  each  person  in  the 
population  has  a  fair  or  equal  chance  of  being  selected.  In  this  way,  a 
sample  that  is  representative  of  the  population  is  obtained. 

Probability  samples  are  used  so  reliable  estimates  of  the  whole 
population  can  be  made  (even  though  not  everyone  in  the  population  is 
part  of  the  survey). 
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3  Design  the  Sampling  Plan  (continued) 


What  are 

non-probability 

samples? 


A  non-probability  sample  uses  human  judgment  in  selecting 
respondents.  Such  judgmental  samples  have  no  theoretical  basis  for 
estimating  population  characteristics.  Non-probability  samples  do  not 
ensure  that  the  sample  is  representative  of  the  population.  These  types 
of  samples  are  referred  to  in  a  number  of  ways,  including 

•  convenience  samples 

•  judgment  samples 

•  self-selecting  sample 


Examples: 

non-probability 

samples 


Examples  of  the  use  of  non-probability  samples  include 

•  distributing  questionnaires  to  participants  at  a  conference  or  event 

•  polling  a  set  of  individuals  considered  to  be  experts  in  an  area  of 
interest 

•  distributing  questionnaires  using  the  Web  and  allowing  individuals 
to  decide  for  themselves  whether  they  want  to  participate  (in  a 
probability  sample,  individuals  are  randomly  selected  to  participate) 

In  the  examples  mentioned  above,  the  findings  will  apply  to  only  those 
who  respond.  It  would  be  incorrect  to  generalize  the  results  to  any 
other  people,  times,  or  places. 

This  is  a  very  important  distinction  and  has  been  a  source  of  confusion 
in  some  surveys  that  have  been  conducted  and  reported  in  the 
literature. 


What’s 
wrong  with 
non-probability 
samples? 


If  the  survey  results  are  intended  to  apply  only  to  a  unique  set  of 
individuals,  it  is  fine  to  distribute  the  questionnaire  and  report  findings, 
as  long  as  it  is  clear  to  readers  of  the  report  that  inferences  and 
generalizations  cannot  be  made  to  those  outside  the  respondent  group. 
In  other  words,  the  reader  of  the  report  should  understand  that  if  the 
questionnaire  were  distributed  to  a  different  set  of  like-individuals,  then 
a  different  set  of  results,  perhaps  in  conflict  with  the  original  sample 
results,  could  be  obtained. 

Unacceptable  uses  of  non-probability  samples 
It  is  not  acceptable  to  draw  a  non-probabilistic  sample  simply  because 
you  are  unwilling  to  spend  the  effort,  time,  or  money  required  to  draw 
a  true  probability  sample. 

When  generalizations  from  the  sample  are  intended  to  apply  to  the 
population,  then  a  probability  sample  must  be  used. 


Is  there  anything  wrong  with  non-probability  samples?  That  depends. 
Acceptable  uses  of  non-probability  samples 

When  survey  findings  are  unique  for  a  particular  set  of  individuals,  a 
sampling  plan  is  not  needed.  Individuals  are  invited  to  participate  in 
the  survey  and  they  do  so  if  they  wish. 
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3  Design  the  Sampling  Plan  (continued) 


Bias  is  introduced  into  surveys  when  a  sample  is  drawn  from  a 
population,  but  the  sample  is  not  representative  of  the  entire 
population.4  Whenever  predispositions  enter  into  the  sampling 
approach,  the  data  is  at  risk  of  becoming  biased.  A  probability  sample 
is  based  on  mathematical  laws  of  chance  rather  than  the  researcher’s 
personal  values  and  judgments. 

Self-selection  can  When  individuals  are  asked  to  volunteer  to  participate  in  a  survey,  bias 
introduce  bias  can  be  introduced.  The  results  obtained  through  this  method  can  be 
misleading.  You  must  question  whether  the  individuals  who  chose  to 
respond  to  the  survey  are  really  representative  of  the  population,  or  if 
one  of  the  following  scenarios  is  at  play: 

•  those  responding  have  a  special  axe  to  grind  so  they  are  taking  the 
time  to  respond  to  the  survey 

•  individuals  who  respond  have  more  slack  time  than  busy  people 
and  the  sample  is  skewed  in  the  direction  of  including  more  non¬ 
busy  individuals  than  those  who  are  over-committed  with 
responsibilities 

In  both  of  these  examples,  there  is  a  high  risk  that  the  results  obtained 
from  self-selected  volunteers  would  not  represent  the  overall  views  of 
the  population  in  which  you  were  interested. 

This  is  why  newscasters  on  television  are  careful  to  report  the  results 
of  call-in  polls  as  non-scientific.  The  people  who  take  the  time  and 
trouble  to  respond  to  these  polls  are  not  likely  to  be  representative  of 
the  population. 


Probability 
samples 
reduce  bias 


Is  there  a  perfect 

probability 

sample? 


No,  having  a  perfect  probability  sample  is  not  possible.  Even  a 
properly  designed  sampling  method  will  often  break  down  somewhat 
due  to  indefinite  populations,  low  response  rates,  or  other  related 
factors.  However,  despite  these  breakdowns,  you  must  make  every 
effort  to  control  the  influence  of  personal  biases  along  the  way  so  you 
will  end  up  as  close  to  a  probability  sampling  design  as  possible. 
Deviations  from  the  planned  sampling  approach  must  be  described  in 
the  survey  report. 


4  This  is  a  special  form  of  bias  known  as  “sampling  bias.”  Other  types  of  bias  can  be  introduced 
through  errors  in  question- writing  and  in  how  results  are  interpreted.  These  issues  are  addressed  in 
substage  4.2.2  beginning  on  page  9  and  stage  7  beginning  on  page  9. 
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3  Design  the  Sampling  Plan  (continued) 


Types  of 

probability 

samples 


Simple  random 
sampling 


Drawing  a 

probability 

sample 


Probability  samples  can  be  obtained  through  a  variety  of  sampling 
methods,  including  the  following: 

•  simple  random  sampling 

•  systemic  sampling 

•  stratified  sampling 

•  cluster  sampling 

The  specific  approach  that  should  be  used  is  determined  by  different 
factors,  including  the  following: 

•  the  size  and  nature  of  the  population 

•  time  and  cost  involved 

•  known  population  characteristics  [Backstrom  81] 


When  the  population  is  a  manageable  size  and  can  be  enumerated, 
simple  random  sampling  is  the  most  straightforward  approach.  Since 
this  is  the  case  for  most  systems  and  software  organizations,  random 
sampling  is  likely  to  be  the  preferred  approach  and  will  be  discussed  in 
this  document. 

Other  approaches  to  probability  sampling  are  variants  of  simple 
random  sampling.  You  can  find  out  more  about  these  approaches  in 
Backstrom  [Backstrom  81]. 


Drawing  a  simple  random  sample  requires  that  all  members  of  the 
population  be  enumerated.  In  many  cases,  doing  this  is  not  a  problem 
since  organizations  maintain  databases  that  list  their  employees. 

Database  lists  can  be  imported  into  common  desktop  spreadsheet 
applications  that  will  assign  random  numbers  to  each  member  of  the 
list.  Many  statistical  software  packages  also  have  this  basic  capability. 

A  simple  way  to  obtain  a  random  sample  using  a  standard  desktop 
spreadsheet  application  is  listed  below. 

1.  List  all  names  associated  with  the  population  under  study. 

2.  Assign  a  random  number  to  each  name  in  an  adjacent  column. 

3.  Sort  the  list  of  names  based  on  the  column  of  random  numbers. 
Select  the  individuals  from  the  list  beginning  at  the  top  until  the 
required  sample  size  has  been  reached. 


24 


CMU/SEI-2005-HB-004 


3  Design  the  Sampling  Plan  (continued) 


What  should  the 
sample  size  be? 


Precision 


Confidence  level 


Thus  far,  the  importance  of  drawing  a  representative  sample  has  been 
discussed,  but  not  how  large  that  sample  should  be. 

The  appropriate  sample  size  can  be  calculated  using  formulas 
described  later  in  this  section.  But,  before  the  equations  are  presented, 
you  should  understand  the  conceptual  underpinnings  of  the 
components  of  the  equations. 

Determining  the  appropriate  sample  size  is  a  function  of 

•  precision  desired 

•  confidence  level  desired 

•  population  size  (when  the  population  size  is  small) 


Precision  is  a  measure  of  how  close  an  estimate  (resulting  from  the 
survey  data)  is  to  the  actual  characteristic  in  the  population.  Precision  is 
related  to  the  error  introduced  by  estimating  characteristics  of  the 
population  through  a  sample.  In  fact,  precision  and  sampling  error  are 
the  flip  sides  of  the  same  coin. 

Sampling  error  =  1-Precision 

Of  course,  all  researchers  want  to  increase  precision.  But,  there  is  a 
price  to  pay.  As  precision  increases,  so  does  the  sample  size — and 
therein  lies  the  trade-off. 

The  level  of  precision  depends  on  the  amount  of  risk  you  are  willing  to 
accept  when  you  use  the  data  to  make  decisions.  You  need  only  the 
level  of  precision  adequate  to  make  the  required  decision  based  on  the 
findings.  In  some  cases,  a  high  degree  of  precision  is  desired.  In  other 
cases,  you  might  need  to  sacrifice  precision  due  to  the  difficulty  of 
obtaining  a  large  sample. 


Sample  size  is  also  a  function  of  confidence  level.  Despite  efforts  to 
draw  a  representative  probability  sample,  there  is  always  some  chance 
of  drawing  a  bad  sample  (i.e.,  one  that  is  not  truly  representative). 
Therefore,  how  confident  can  you  be  that  the  sample  is  a  good  one, 
and  truly  represents  the  population? 

The  confidence  level  tells  you  how  confident  you  can  be  that  the  error 
toleration  does  not  exceed  what  was  planned  for  in  the  precision 
specification.  To  understand  confidence  levels,  think  of  drawing  100 
independent  samples  from  the  population.  How  sure  do  you  want  to  be 
that  the  sample  is  a  good  one?  Specifying  a  95%  confidence  level 
would  imply  that  if  we  took  100  independent  samples,  then  95  out  of 
the  100  would  be  within  the  error  tolerances  specified  by  the  precision. 
If  you  have  a  99%  confidence  level,  then  you  can  feel  confident  that 
99  out  of  the  100  samples  would  provide  an  estimate  within  the 
precision  set  by  the  researcher. 

Continued  on  next  page 
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3  Design  the  Sampling  Plan  (continued) 


Confidence  level 
(continued) 


The  confidence  level  is  ascertained  through  the  well-established 
probability  model  called  the  Standard  Normal  Distribution  and  an 
associated  theorem  called  the  Central  Limit  Theorem  [Feller  68, 
Kallenberg  97,  Trotter  59],  The  theory  behind  the  use  of  the  standard 
normal  distribution  is  taught  in  basic  statistics  courses  (and  courses 
sometimes  taught  in  secondary  school  mathematics). 

When  calculating  the  sample  size,  the  desired  confidence  level  is 
specified  by  the  z-value.  The  z-value  is  a  point  along  the  abscissa  of 
the  standard  normal  distribution,  as  shown  below. 


Standard  Normal  Distribution 


z  values 


A  given  value  for  z  specifies  an  area  under  the  standard  normal  curve. 
The  figure  below  illustrates  how  a  z-value  of  1.96  specifies  95%  of  the 
area  under  the  normal  curve.  This  would  represent  a  confidence  level 
of  95%.  A  z-value  of  2.58  would  specify  99%  of  the  area  under  the 
normal  curve.  In  this  present  context,  this  z-value  would  specify  a 
confidence  level  of  99%. 


Standard  Normal  Distribution 


Values  for  z  and  corresponding  normal  curve  areas  can  be  found  in 
tables  supplied  in  the  back  of  most  statistic  books.  The  function  is  also 
available  in  most  spreadsheet  applications  and  in  all  specialized 
statistical  applications. 
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3  Design  the  Sampling  Plan  (continued) 


Population 

size 


Important 

assumptions 


When  your 
population 
is  large 


The  size  of  your  population  can  also  affect  your  sample  size 
calculation,  but  only  when  your  population  becomes  small. 

There  are  two  different  formulas  for  calculating  the  sample  size 
depending  on  whether  your  population  is  large  or  small.  These  are 
shown  later  in  this  section. 


Note  that  sample  size  calculations  are  based  on  the  following 
assumptions: 

•  The  sample  is  obtained  through  simple  random  sampling 
(see  page  24). 

•  The  sample  sizes  obtained  from  the  formulas  in  this  section  reflect 
the  number  of  obtained  responses,  not  the  number  of  requests  for 
questionnaire  responses.  Quite  often,  the  sample  size  used  in  the 
survey  is  increased  by  30%  to  compensate  for  non-response  so  that 
the  power  of  the  survey  (i.e.,  precision  and  confidence  level)  is  not 
compromised  [Israel  92]. 


To  calculate  your  sample  size,  you  must  specify  the  precision  and 
the  confidence  interval  you  are  willing  to  accept.  Again,  the  values 
you  specify  are  based  on  the  amount  of  risk  you  are  willing  to  accept 
that  the  sample  size  would  be  a  poor  representation  of  the  population. 
(See  the  description  of  precision  and  confidence  level  earlier  in  this 
section.) 


Cochran  developed  a  formula  for  calculating  the  sample  size  from  a 
large  population  [Cochran  77]. 


no  = 


Z2pq 


where 

n„ 


is  the  sample  size 


(1) 


Z  is  a  point  on  the  abscissa  of  the  standard  normal  curve  that 
specifies  the  confidence  level  (see  the  diagram  on  page  26). 
p  is  an  estimated  proportion  of  an  attribute  that  is  present  in  the 
population 
q  is  equal  to  (1-p) 

e  specifies  the  desired  level  of  precision  where 
e  =  1  -  precision 


The  mathematical  value  (p*q )  is  an  estimate  of  the  variance  of  the 
population  attribute  (assumed  to  be  dichotomous)  that  you  are  trying 
to  measure  with  your  survey.  Of  course,  there  is  no  way  to  really  know 
what  this  is  before  measuring  it.  Therefore,  for  sample  size 
calculations,  this  value  is  always  assumed  to  be  the  maximum 

variance.  -  _ 

Continued  on  next  page 
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3  Design  the  Sampling  Plan  (continued) 


When  your 
population 
is  large 
(continued) 


The  maximum  variance  is  represented  by  the  following  equation: 

pq  =  0.5  *0.5  =  0.25 


This  equation  represents  a  case  where  a  questionnaire  response 
attribute  is  evenly  split  among  the  population. 


You  might  be  wondering  why  the  calculation  assumes  a  dichotomous 
response  variable  (e.g.,  a  response  variable  that  is  “yes”  or  “no”). 

What  about  those  questions  that  have  a  more  continuous-type  response 
scale  (e.g.,  a  Likert-type  scale)?  In  those  case,  the  sample  size  equation 
would  be 


The  above  formula  employs  a,  the  population  variance,  instead  of 
( p*q )  as  the  measure  of  variability.  The  disadvantage  here  is  that 
establishing  a  good  estimate  of  the  population  variance  is  necessary  to 
use  this  formula. 


Example: 
sample  size  for 
large  population 


Consider  an  example  where  the  population  size  is  8,750  and  you 
would  like  to  draw  a  representative  sample  with  precision  of  95%  and 
a  confidence  level  of  99%.  The  sample  size  would  be  expressed  as 


«o 


z2pq  _  2.582(0.5  *  0.5) 
e2  0.052 


=  665.6  *  666 


Does  size 
matter  when 
population  is 
very  large? 


Although  it  might  seem  counterintuitive  to  those  without  a  statistics 
background,  population  size  becomes  irrelevant  in  surveys  because  the 
same  mathematical  principles  of  random  sampling  apply  to  large  and 
not-so-large  populations. 

For  the  same  level  of  precision  and  confidence,  a  sample  of  about  395 
people  will  describe  a  population  of  100,000  people  just  as  well  as  it 
will  describe  a  population  of  10,000  people.  The 
best  way  to  think  about  this  is  through  an  analogy 
used  for  sampling: 

Every  cook  knows  that  it  only  takes  a  single  sip  from 
a  well-stirred  soup  to  determine  the  taste. 

This  is  a  good  analogy  for  sampling  because  it  emphasizes  the  need 
for  the  soup  to  be  well-stirred  to  produce  the  random  sample  required 
if  the  sample  is  to  represent  the  population. 

Now,  using  this  analogy  again,  the  sip  (sample)  would  determine  the 
taste  whether  it  was  from  a  large  pot  or  a  small  one.  The  idea  here  is 
that  the  size  of  the  soup  pot  doesn’t  really  matter.  The  sample  (sip) 
represents  the  population  (the  pot  of  soup)  whether  the  population  is 
big  or  small. 
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3  Design  the  Sampling  Plan  (continued) 


When  your 
population 
is  small 


The  analogy  just  described  only  goes  so  far.  When  the  population  size 
is  significantly  small,  then  the  size  of  the  pot  does  begin  to  matter. 


When  your  population  size  is  small,  you  can  employ  the  finite 
population  correction  (fpc)  factor  [Cochran  77,  Kish  65].  The  finite 
population  correction  factor  is: 


fpc  = 


\N_Zlk 

N- 1 


(3) 


where 

N  is  the  population  size 

n0  is  the  sample  size  (as  calculated  in  equation  #1  on  page 

27). 


The  finite  population  correction  factor  measures  how  much  extra 
precision  is  achieved  when  the  sample  size  becomes  close  to  the 
population  size. 


The  sample  size  derived  by  using  the  finite  population  correction 
factor  will  be  smaller  than  that  derived  from  the  uncorrected  equation. 

By  using  the  fpc,  a  revised  sample  size  can  be  calculated  using  the 
following  formula: 


where: 


nR  = 


n 


o 


1  + 


K-i) 

N 


(4) 


nR  is  the  revised  sample  size  based  on  the  fpc 
N  is  the  population  size 

no  is  the  sample  size  (as  calculated  in  equation  1  on  page  27) 


When  can  you  use  The  fpc  can  be  used  when  the  sample  size  exceeds  10%  of  the 

the  fpc?  population  size  [Kish  65]. 
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3  Design  the  Sampling  Plan  (continued) 


Example: 
using 
the  f pc 


Consider  the  case  where: 


Population  size  =  2000 
Desired  precision  =  95% 
Confidence  interval  =  95% 

Using  equation  1,  the  sample  size  is: 

1.962(0.5  *  0.5) 


«o  = 


Z2pq 


=  384.1 


e2  0.05 

Since  384/2000  =  19%  and  19%  is  greater  than  the 
criterion  of  10%,  using  the  fpc  is  permitted. 


«  384 

fpc  threshold 


Therefore,  the  revised  sample  size  number  is 


(n.~D 

N 


384 

n  (384-1) 
2000 


=  322.3  *  322 


Therefore,  in  this  case,  you  only  need  to  sample  322  individuals 
(rather  than  384)  to  obtain  the  same  level  of  precision  and  confidence. 


A  simplified 
formula  for 
sample  size 


Yamane  provides  a  simplified  formula  to  calculate  sample  sizes.  This 
simplified  formula  assumes  a  95%  confidence  level  and  the  maximum 
variance  (p  =  0.5).  The  formula  is 


N 

n  = - - 

1  +  iV(e)2 

where 


n  is  the  sample  size. 

N  is  the  population  size 

e  specifies  the  desired  level  of  precision,  where 
e  =  1  -  precision 

The  table  on  the  next  page  lists  sample  sizes  for  various  precision 
levels  using  Yamane’s  formula  [Yamane  67]. 
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3  Design  the  Sampling  Plan  (continued) 


Sample  sizes 
based  on  the 
Yamane  formula 


The  following  table  presents  sample  size  numbers  for 
various  levels  of  precision  and  a  confidence  level  of  95% 
with  p  =  0.5  (maximum  variance). 


Precision  Level 

Population  Size 

±3% 

±5% 

±7% 

±10% 

100 

* 

80 

67 

50 

125 

* 

95 

78 

56 

150 

* 

109 

86 

60 

175 

* 

122 

94 

64 

200 

* 

133 

101 

67 

225 

* 

144 

107 

69 

250 

* 

154 

112 

71 

275 

* 

163 

117 

73 

300 

* 

171 

121 

75 

500 

* 

222 

145 

83 

600 

* 

240 

152 

86 

700 

* 

255 

158 

88 

800 

* 

267 

163 

89 

900 

* 

277 

166 

90 

1,000 

* 

286 

169 

91 

2,000 

714 

333 

185 

95 

3,000 

811 

353 

191 

97 

4,000 

870 

364 

194 

98 

5,000 

909 

370 

196 

98 

6,000 

938 

375 

197 

98 

7,000 

959 

378 

198 

99 

8,000 

976 

381 

199 

99 

9,000 

989 

383 

200 

99 

10,000 

1,000 

385 

200 

99 

15,000 

1,034 

390 

201 

99 

20,000 

1,053 

392 

202 

100 

25,000 

1,064 

394 

202 

100 

50,000 

1,087 

397 

203 

100 

100,000 

1,099 

398 

204 

100 

>100000 

1,111 

400 

204 

100 

For  cells  containing  an  asterisk,  the  assumption  of  statistical  normality  is 
poor  and  the  entire  population  should  be  sampled  [Yamane  67}. 
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3  Design  the  Sampling  Plan  (continued) 


Very  small 
populations 


When  your  population  size  is  very  small  (less  than  200  individuals), 
you  should  consider  conducting  a  census  [Kish  65].  A  “census”  is  a 
sample  that  includes  all  individuals  in  the  population  in  the  sample.  A 
census  eliminates  sampling  error  and  provides  information  on  all 
individuals  in  the  population.  For  samples  of  200  or  less,  the  entire 
population  has  to  be  included  in  the  sample  to  achieve  a  desirable  level 
of  precision. 
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4  Design  and  Write  the  Questionnaire — 
Overview 


Introduction  In  this  section  the  issues  associated  with  the  design  and  development 

of  the  questionnaire  instrument  are  discussed.  This  stage  is  divided 
into  substages  4.1  and  4.2,  as  illustrated  in  the  diagram  above.  These 
substages  are  represented  as  linear,  but  there  will  likely  be  several 
iterations  between  them  as  the  instrument  is  developed.  Each  substage  is 
covered  in  a  separate  subsection. 


Key  inputs  In  a  way,  the  questionnaire  design  and  writing  started  during  stages  1 

and  2  when  the  research  objectives  were  defined  and  the  target 
audience  was  identified  and  characterized.  The  products  of  these  two 
stages  provide  important  input  to  how  the  survey  should  be  designed 
and  written. 


1.  Identify  the  research 
objectives 


2.  Identify  &  characterize 
target  audience 


Objectives 


f  4.  Design  &  write 
questionnaire 


Audience  characteristics 
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4  Design  and  Write  the  Questionnaire — Overview  (continued) 


Why  objectives 
are  important 


Why  audience 
characteristics 
are  important 


Designing  the 
survey 

questionnaire 


The  importance  of  well-defined  objectives  (stage  1)  can  not  be  over¬ 
emphasized.  A  questionnaire  that  is  written  without  clear  objectives  will 
overlook  important  issues  and  waste  respondents’  time  by  asking 
useless  questions. 

The  problems  of  a  poorly  defined  questionnaire  continue  on  to  the 
analysis  stage  (stage  7).  It  is  difficult  to  imagine  identifying  a  problem 
and  its  cause,  let  alone  its  solution,  from  responses  to  broad  and 
general  questions. 

In  other  words,  how  can  you  reach  insightful  conclusions  if  you  do  not 
know  what  you  were  looking  for  or  planning  to  observe? 

If  you  are  finding  it  difficult  to  write  questions,  then  perhaps  you  have  not 
spent  enough  time  defining  the  objectives  of  the  survey.  Go  back  to  that 
stage  again.  Questions  should  follow  naturally  from  well-written 
objectives. 


One  of  the  biggest  mistakes  novices  make  is  writing  a  questionnaire 
for  themselves.  That  is,  they  use  language  familiar  to  themselves  as 
researchers  but  unintelligible  to  those  who  will  be  responding  to  the 
questionnaire  items.  This  is  especially  the  case  in  the  rather  immature 
domain  of  systems  and  software  engineering,  where  terms  are 
sometimes  ill-defined.  Different  respondents  interpret  them  differently 
based  on  their  cultures,  experiences,  and  knowledge. 

A  questionnaire  must  be  written  with  the  target  audience  in  mind.  The 
audience  analysis  conducted  during  stage  2  is  used  to  help  the 
researcher  develop  understandable  questionnaire  items. 


Questionnaire  design  addresses  the  questions  listed  below. 

•  How  will  the  survey  be  mediated  (e.g.,  via  paper,  email  soft  copy, 
Web)? 

•  How  long  should  the  questionnaire  be? 

•  How  should  the  questionnaire  be  structured  and  organized? 

•  What  page  design  and  formatting  will  be  most  effective? 

•  What  other  documents  might  promote  survey  effectiveness? 
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4  Design  and  Write  the  Questionnaire — Overview  (continued) 


Writing  the 
questionnaire 


In  this  section 


There  are  many  things  to  consider  when  writing  the  questionnaire, 

including 

•  Do  you  really  know  what  you  want  to  measure? 

•  Are  you  writing  the  questions  so  they  will  be  understandable  to  all 
members  of  your  intended  audience? 

•  How  complex  are  the  questions  you  need  to  have  answered?  How 
difficult  will  it  be  for  respondents  to  answer  the  questions? 

•  Will  respondents  be  able  and  willing  to  provide  accurate  answers? 

•  Are  you  asking  the  questions  in  a  way  that  allows  the  results  to  be 
easily  tabulated  and  measured? 


Topic 

See  page 

Determine  the  questions  to  be  asked 

36 

Select  the  question  type,  format  and  specific  wording 

40 

Design  the  question  sequence  and  overall 

54 

questionnaire  layout 

Develop  ancillary  documents 

71 
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4.1 


Determine  Questions  to  be  Asked 


—  4.1  Determine  the  questions  to  be  asked 

—  4.2  Select  the  question  type/format  for  each  question  and 

specify  wording 

—  4.3  Design  the  question  sequence  and  overall  questionnaire  layout 

—  4.4  Develop  ancillary  documents 


Introduction  Distinguishing  between  the  questions  that  the  research  team  need 

answered  (i.e.,  the  internal  questions)  and  the  questions  a  respondent  is 
to  answer  (the  questionnaire  items)  is  very  important.  While  these  two 
are  related,  the  questions  the  research  team  has  are  usually  unsuitable  for 
asking  respondents. 

In  this  section  we  discuss  formulating  the  internal  questions  based  on 
the  objectives  identified  during  stage  1.  During  this  stage,  don’t  be 
overly  concerned  about  wording  and  format  issues  as  long  as  the 
questions  are  clear  and  align  to  the  survey  objectives.  These  questions 
will  then  be  massaged  into  appropriate  questionnaire  items  during 
substage  4.2. 


Types  of  There  are  four  main  types  of  questions  you  might  want  to  use  in  your 

questions  survey.  You  can  ask  people  about 

•  attributes 

•  attitudes 

•  beliefs 

•  behaviors 

It  is  important  to  distinguish  among  these  types  of  information. 
Otherwise,  attempts  to  reword  questions  during  substage  4.2  might 
unwittingly  lead  not  only  to  a  change  in  wording,  but  a  complete  change 
in  the  kind  of  information  being  elicited. 
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4.1  Determine  Questions  to  be  Asked  (continued) 


Questions  about 
attributes 


Questions  about 
attitudes 


Questions  about 
beliefs 


Beliefs  vs. 
attitudes 


Questions  about 
behaviors 


Generating  the 
questions: 
a  team  activity 


Attribute  questions  typically  ask  for  information  about  personal  or 
demographic  characteristics.  Virtually  all  surveys  ask  about  a  number 
of  these  attributes.  Among  the  most  frequently  requested  attributes  are 
age,  education,  organizational  size,  occupation,  years  of  experience  (at 
a  particular  occupation),  and  title  or  rank  (if  military). 

Researchers  collect  this  type  of  information  so  they  can  explore  how 
questions  about  beliefs,  attitudes,  and  behavior  differ  for  respondents 
with  various  attributes. 


Questions  about  attitudes  differ  from  questions  about  beliefs  in  that 
they  tap  into  the  respondent’s  personal  outlook  and  orientation,  which 
have  been  acquired  through  years  of  experience.  Questions  about 
attitude  ask  people  how  they  feel  about  something  and  require  them  to 
indicate  whether  they  have  positive  or  negative  feelings  about  a  topic. 


Beliefs  are  the  respondents’  assessments  of  what  they  think  is  true  or 
false.  These  questions,  also  referred  to  as  opinion  questions,  try  to 
determine  how  people  think  about  a  particular  subject  at  a  particular 
time.  Belief  questions  are  more  focused  and  specific  than  attitude 
questions. 


The  distinction  between  belief  questions  and  attitude  questions  is 
sometimes  a  gray  area.  However,  the  conceptual  distinction  is  important 
and  can  guide  the  rewording  of  questions  during  substage  4.2. 


Behavior  questions  ask  respondents  to  describe  what  they  have  done  in 
the  past  or  what  they  are  currently  doing.  They  are  typically  analyzed 
in  terms  of 

•  whether  the  behavior  is  present  or  absent 

•  frequency  of  the  behavior 

•  degree  of  behavior  performance  completion 

Strictly  speaking,  behavior  questions  elicit  respondents’  beliefs  about 
their  behaviors.  However,  the  distinction  is  useful  because  people  asked 
to  describe  their  behavior  often  answer  in  a  different  way  than  they  do 
when  asked  about  their  experience  of  something  in  a  cognitive  way. 


As  discussed  on  page  9,  survey  work  should  employ  a  team  approach. 
If  you  are  the  researcher,  do  yourself  a  favor  and  do  not  assume  sole 
responsibility  for  generating  the  internal  questions.  Your  primary  task 
during  this  substage  is  to  facilitate  and  elicit  these  questions  from  other 
team  members. 
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4.1  Determine  Questions  to  be  Asked  (continued) 


Reviews  &  approves 

»  •  • 

Management 


If 


& A  &  Draft 

1 11  questions 

i 

* 

kjl 

Substage  4.2 


Subiect  Matter  Experts 


Facilitates  question  input 
&  refines  draft  questions 


Researcher 


Questionnaire 

Instrument 

Approach  Below  is  a  team  approach  you  can  use  to  generate  internal  questions. 


Step 

Who 

Description 

1 

Researcher 

Distribute  a  list  of  the  survey  objectives  to 
the  team  members  asking  them  to  formulate 
their  questions  of  interest  against  each 
objective. 

2 

Subject 

matter 

experts 

(SMEs) 

For  each  objective,  write  questions  you 
believe  are  relevant.  Send  these  questions  to 
the  researcher. 

3 

Researcher 

Combine  questions  from  stage  2  into  a  single 
list.  Organize  questions  for  each  objective. 

If  some  questions  are  not  mapped  to 
objectives,  note  this  and  consider  whether 
the  list  of  objectives  needs  to  be  expanded. 

4 

Researcher 

Organize  a  meeting  to  review  the  submitted 
draft  questions.  Send  the  list  to  SMEs  before 
the  meeting  and  ask  them  to  be  prepared  to 
prioritize  the  list  of  questions. 

5 

Team 

Conduct  a  review  meeting.  The  researcher 
facilitates  while  the  team  discusses  each 
question.  Clarifications  are  sought  where 
needed.  All  unusual  terminology  is  defined  as 
a  part  of  this  team  exercise. 

The  team  then  uses  a  structured  group  technique 
such  as  multivoting5  to  prioritize  the  list  of 
questions. 

38 


See  the  discussion  of  multivoting  in  The  Team  Handbook  by  Peter  R.  Scholtes.  Joiner  Associates 
Inc.,  1988,  ISBN  0-9622264-0-8. 


CMU/SEI-2005-HB-004 


4.1  Determine  Questions  to  be  Asked  (continued) 


Results 


Why  prioritize 
questions? 


As  a  result  of  this  substage,  the  researcher  should  have  a  list  of 
prioritized  questions  that  represents  the  team’s  thinking  about  the 
information  to  be  sought  through  the  survey. 


As  part  of  the  above  approach,  the  team  prioritized  the  question  list 
during  step  4.  Prioritization  is  typically  needed  when  the  team  has 
many  questions  they  want  to  gather  information  about.  There  are 
practical  limitations  on  the  size  of  a  questionnaire  instrument.  In  many 
cases,  difficult  decisions  must  be  made  to  eliminate  questions  from  the 
questionnaire  simply  because  the  length  must  be  such  that  it  can  be 
completed  within  a  reasonable  amount  of  time.  This  will  be  discussed 
next  in  subsection  4.2. 
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4.2  Select  Question  Type,  Format,  and  Specific 
Wording — Overview 


—  4.1  Determine  the  questions  to  be  asked 

—  4.2  Select  the  question  type/format  tor  each  question  and 

specify  wording 

—  4.3  Design  the  question  sequence  and  overall  questionnaire  layout 

—  4.4  Develop  ancillary  documents 

Introduction  During  this  substage,  the  researcher  takes  the  output  from  substage  4. 1 

(i.e.,  the  team’s  internal  questions)  to  develop  the  actual  survey 
questionnaire.  The  questions  are  rewritten  so  that: 

•  the  responses  can  be  quantitatively  analyzed  (question  structure 
and  format) 

•  they  are  understandable  to  the  target  audience  (question  wording) 

Rewriting  for  Questions  must  be  rewritten  so  that  the  responses  can  be  quantitatively 

response  analysis  assessed.  This  is  usually  done  by  transforming  open-ended  internal 

questions  into  closed-ended  questions.  In  many  cases,  multiple  closed- 
ended  questions  will  be  required  to  address  a  single  open-ended 
internal  question. 

Open-ended  questions  are  those  that  ask  respondents  to  express 
themselves  freely  without  any  constraints.  These  questions  have  no 
answer  choices  from  which  respondents  select  their  responses.  Instead, 
respondents  create  their  own  answers  and  state  them  in  their  own 
words. 

Closed-ended  questions  are  questions  that  provide  answer  choices. 
Respondents  choose  from  among  the  provided  answer  categories  by 
evaluating  each  choice  and  selecting  the  one  that  best  reflects  their 
situations.  Variations  of  closed-ended  questions  will  be  discussed  in 
section  4.2  on  page  40. 

One  disadvantage  of  using  open-ended  questions  in  self-administered 
questionnaires  is  the  difficultly  of  constructing  meaningful  variables 
for  statistical  analysis.  Therefore,  most  internal  questions  (developed 
during  substage  4.1)  should  be  transformed  into  closed-ended 
questions. 
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4.2  Select  Question  Type,  Format,  and  Specific  Wording — Overview  (continued) 


Rewriting  for  The  wording  of  questions  is  extremely  important.  The  questions  you 

understandability  and  your  team  members  ask  yourselves  are  almost  always  unsuitable 
for  asking  respondents  directly.  Writing  questions  for  themselves  is 
one  of  the  main  mistakes  novices  make  when  writing  questionnaires. 
The  next  section  addresses  this  problem  in  more  detail. 


Comment  For  the  sake  of  clarity,  the  discussions  about  the  question  structure  and 

format  (section  4.2.1)  and  question  wording  (section  4.2.2)  are 
discussed  separately.  However,  the  constructs  presented  in  each 
section  should  not  be  applied  in  a  linear  fashion  but  considered 
concurrently  as  the  internal  questions  are  translated  into  questionnaire 
items  for  respondents. 


Reminder  Remember  that  better  survey  instruments  are  the  product  of  a  team 

approach.  The  researcher  should  schedule  appropriate  reviews  as  the 
survey  questionnaire  is  being  developed. 


Internal 

questions 


Transformed 

„  .  .  .  _  questions 

Substage  4.2  - 


/  JJ 

y  t  # 


Subject  Matter  Experts 


\ 


If 


Management 


Researcher 


Transforms  internal  questions  using 
skills  described  in  substage  4.2. 1 
and  substage  4.2.2 


Reviews  and  provides 
feedback  on  intermediate 
work  products 


In  this 
section 


: 

Section 

See  page 

Question  structure  and  response  format 

42 

Question  wording 

48 

CMU/SEI-2005-HB-004 


41 


4.2.1  Question  Structure  and  Response  Format 


Introduction 


Question 
structure  & 
response  format: 
what  are  they? 


Types  of 
question 
structure 


A  key  and  powerful  property  of  surveys  is  their  ability  to  characterize 
the  opinions  and  behaviors  of  a  population  quantitatively  in  a  way  that 
permits  uniform  interpretation  of  the  population’s  attributes. 


Question  structure  refers  to  the  nature  of  the  response  behavior  that  is 
being  asked  of  the  respondent.  There  are  two  main  categories: 

•  open-ended  questions  (sometimes  referred  to  as  unstructured 
questions) 

•  closed-ended  questions  (sometimes  referred  to  as  structured 
questions) 

In  this  section  we  will  discuss  these  two  categories  and  variations  of  each. 

Response  format  refers  to  how  you  collect  the  answer  from  the 
respondent.  A  number  of  response  formats  are  associated  with  closed- 
ended  questions.  Some  of  the  most  commonly  used  formats  will  be 
discussed  in  this  section.  The  response  format  for  open-ended 
questions  is  simply  written  text. 


How  to  structure  questions  is  a  major  decision  for  the  researcher.  The  four 
different  question  structures  discussed  in  this  document  are  described  in 
the  table  below. 


Structure 

Description 

Open-ended 

Respondents  create  their  own  answers  to  the  question 
in  their  own  words.  There  are  no  answer  choices 
provided  from  which  they  select  their  responses. 

Closed-ended 

with 

unordered 

choices 

Respondents  choose  from  among  discrete,  unordered 
categories  by  evaluating  each  choice  and  selecting  the 
most  appropriate  response  or  responses. 

Closed-ended 

with  ordered 

choices 

Answer  choices  are  provided  and  the  choices  are 
represented  as  a  graduated  scale  along  some  single 
dimension  of  opinion  or  behavior.  The  respondent 
chooses  the  most  appropriate  response  from  the 
choices  provided. 

Hybrid 

Answer  choices  are  provided  (as  in  closed-ended 
questions),  but  the  respondent  can  also  create  a 
response  if  the  choices  are  inappropriate  (as  in  an 
open-ended  question). 
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4.2.1  Question  Structure  and  Response  Format  (continued) 


How  will  you 
analyze  the 
results? 


The  topics  covered  in  this  section  are  related  to  those  discussed  in 
subsection  7.1,  “Analyze  the  Results,”  which  begins  on  page  84.  They 
are  related  because  decisions  you  make  (with  regard  to  question 
structure  and  format)  will  impact  how  you  will  be  able  to  analyze  the 
resultant  response  information. 

If  you  are  currently  in  the  throes  of  developing  questionnaire  items, 
then  you  should  read  the  material  in  subsection  7.1  after  you  read  this 
section.  Both  sections  together  present  a  more  complete  view  of  the 
issues  involved  in  selecting  the  proper  question  structure  and  format. 


Open-ended  Open-ended  questions  lead  to  narrative,  word-based  responses. 

questions  lead 

to  qualitative  Qualitative  research  is  a  valid  and  useful  tool  that  can  lead  to  rich  and 

responses  detailed  response  information  about  a  topic  in  the  original  language  of 

the  respondent.  However,  such  data  cannot  be  generalized.  The 
information  is  raw  and  seldom  pre-categorized.  Therefore,  you  must 
apply  very  time-consuming  methods  to  interpret,  organize,  and 
categorize  it. 


Internal  questions  Typically,  internal  questions  are  expressed  as  open-ended  questions 
from  stage  4.1  and  need  to  be  transformed  into  closed-ended  questions.  This  allows 
the  population  to  be  characterized  quantitatively  through  descriptive 
statistics. 

In  some  cases,  an  internal  question  will  be  broad  in  nature  or  worded 
in  such  a  way  that  more  than  one  question  is  being  asked.  When  this  is 
the  case,  the  question  will  need  to  be  transformed  into  more  than  one 
survey  question. 


Can  open-ended 
questions  be  used 
in  surveys? 


Open-ended  questions  are  most  suitable  for  exploratory,  interviewer- 
led  surveys.  However,  if  you  intend  to  include  open-ended  questions, 
they  should  be  used  sparingly — especially  when  the  sample  size  is 
large.  If  you  use  them,  be  prepared  to  allocate  a  significant  amount  of 
time  to  organizing,  interpreting,  filtering,  and  analyzing  the 
information.6  Responses  to  open-ended  questions  can  be  difficult  or 
impossible  to  interpret,  and  since  follow-up  questions  are  not  possible 
in  self-administered  questionnaires,  this  presents  a  problem  of  validity. 
Because  human  judgment  is  required  to  filter  and  interpret  the 
responses,  misinterpretation  is  always  a  risk. 


6  See  on  page  9  of  this-document  for  an  approach  to  accomplishing  this. 
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4.2.1  Question  Structure  and  Response  Format  (continued) 


Transforming 
open-ended 
questions  into 
closed-ended 
questions 


Qualitative  data  consists  of  words,  while  quantitative  data  consists  of 
numbers.  While  this  difference  seems  to  be  significant,  there  is  really 
not  much  difference  between  the  two.  All  qualitative  data  can  be  coded 
and  transformed  into  quantitative  data. 

This  is  done  by  transforming  open-ended  questions  into  closed-ended 
questions  that  lead  to  responses  that  can  be  quantified  and  generalized 
across  the  sample  of  respondents. 

There  are  two  types  of  closed-ended  questions: 

•  closed-ended  questions  with  ordered  responses 

•  closed-ended  questions  with  unordered  responses 


Closed-ended 
questions  with 
ordered 
responses 


Open-ended  questions  can  be  translated  into  closed-ended  questions 
with  ordered  responses  through  the  process  of  scaling.7 

Scaling  is  the  assignment  of  objects  to  numbers  according  to  a  rule 
[Stevens  59].  In  survey  research,  the  objects  are  text  statements — 
especially  statements  of  opinion,  attitude,  or  belief. 

There  are  different  types  of  response  scales,  including 

•  dichotomous  response  scales  (such  as  yes/no,  agree/disagree) 

•  ordinal  response  scales  (e.g.,  poor,  fair,  neutral,  good,  very  good) 

For  the  latter,  each  choice  offered  for  a  response  represents  a  gradation 
of  a  single  dimension  of  some  topic.  This  type  of  question  is  well- 
suited  for  determining  the  intensity  of  a  belief  or  feeling  and  the 
frequency  of  a  behavior. 

Respondents  must  identify  the  response  that  places  them  at  the  most 
appropriate  point  on  a  scale  implied  by  the  answer  choices. 

Ordinal  response  scales  as  described  above  are  commonly  referred  to  as 
“Likert  scales”  due  to  their  association  with  the  Likert  scaling  method 
[Likert  32], 


7  There  are  three  different  types  of  uni-dimensional  scaling  methods  including  Thurstone,  Guttman, 
and  Likert  scaling.  A  discussion  of  these  is  beyond  the  scope  of  this  document.  However,  Trochim 
provides  a  cogent  discussion  of  each  method  [Trochim  01]. 
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4.2.1  Question  Structure  and  Response  Format  (continued) 


Example:  Likert  Below  is  one  example  of  an  ordinal  response  scale. 


response  scale 

Response  Scale 


Strongly  disagree  1 

Disagree  2 

Undecided  3 

Agree  .  4 

Strongly  Agree  5 


Likert  scales  can  have  different  ranges,  for  example  from  0-4, 1-7,  1-9, 
and  so  on.  In  some  cases,  the  scale  is  odd-numbered  with  the  middle 
value  labeled  as  “undecided”  or  “neutral.”  When  there  is  a  possibility 
that  a  respondent  cannot  answer  the  question,  then  it  is  appropriate  to 
include  a  “Don’t  Know”  or  “Doesn’t  Apply”  response  that  is  assigned 
the  scale  value  of  zero. 


Other  exaples  of  See  Appendix  D  beginning  on  page  123  for  examples  of  popular  Likert 

Likert  scales  scales. 


Examples: 
Questions  using 
Likert  response 
scale 


Here  are  some  examples  of  questions  that  employ  a  Likert  response  scale. 


Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

N/A 

Technical  reviewers  use 

proposal  evaluation  criteria 
during  solicitation  activities. 

□ 

□ 

□ 

□ 

□ 

Notice  that  although  numerical  values  are  applied  for  each  response,  the 
respondent  does  not  need  to  be  aware  of  this  assignment  when  completing  the 
questionnaire. 


Closed-ended 
questions  with 
unordered 
responses 


Closed-ended  questions  with  unordered  responses  differ  from  those 
with  ordered  responses.  Ordered  responses  do  not  limit  the  respondents 
to  choosing  from  a  gradation  of  a  one-dimensional  concept.  These  types 
of  questions  provide  unordered  answer  choices  that  respondents  must 
evaluate  in  relation  to  each  other. 

These  types  of  questions  may  also  permit  the  respondent  to  answer  by 
checking  all  responses  that  apply. 
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4.2.1  Question  Structure  and  Response  Format  (continued) 


Example 


Hybrid 

questions 


Example: 

hybrid 

question 


Below  is  an  example  of  a  closed-ended  question  with  unordered  responses. 

Which  of  the  following  have  you  spent  the  most  time  doing  during  the  past 
work  week? 

□  Talking  to  customers  on  the  phone 

□  Attending  meetings 

□  Working  on  my  computer 

□  None  of  the  above 


A  hybrid  question  can  be  thought  of  as  a  partially  closed-ended 
question  that  allows  the  respondents  to  write  their  own  answers  if  the 
options  provided  are  unsuitable.  This  format  does  not  force 
respondents  into  choosing  answers  that  do  not  fit. 

Hybrid  questions  should  always  be  used  when  you  suspect  that  some 
response  options  have  been  overlooked.  They  are  most  appropriate  for 
closed-ended  questions  with  unordered  responses  (rather  than  with 
ordered  responses). 


Below  is  an  example  of  a  hybrid  question. 

Which  of  the  following  have  you  spent  the  most  time  doing  during  the 
past  work  week? 

□  Talking  to  customers  on  the  phone 

□  Attending  meetings 

□  Working  on  my  computer 

□  None  of  the  above 

□  Other  [Please  specify  in  the  space  provided] 


Note  that  if  “other”  is  checked,  respondents  create  their  own  responses 
to  the  question. 
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4.2.1  Question  Structure  and  Response  Format  (continued) 


Section 

summary 


In  this  section,  issues  involving  question  structure  and  response  format 
were  discussed.  Internal  questions  (those  posed  by  you  or  the  subject 
matter  experts  on  your  research  team)  are  typically  open-ended 
questions  that  are  unsuitable  for  a  survey  instrument. 

Also  discussed  were  ways  to  restate  these  open-ended  questions  in 
closed-ended  form  to  facilitate  the  interpretation  and  analysis  of  the 
information  received.  The  next  section  focuses  on  the  word  content  of 
the  questions. 

For  large-sample  surveys,  using  open-ended  questions  might  provide  a 
lot  of  interesting  information,  but  the  task  of  interpreting  and 
organizing  the  response  information  is  daunting  to  say  the  least.  It  is 
very  difficult  to  transform  such  information  into  quantitative  data  that 
can  inform  decision-makers.  For  this  reason,  open-ended  questions 
should  be  used  sparingly  in  survey  instruments. 
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4.2.2  Question  Wording 


Introduction 


The  problem  with 
simple  guidelines 
for  wording 


Earlier  in  this  document,  the  concept  of  construct  validity  was 
introduced  as  an  important  concern  of  the  research  team  (see  page  12). 
Construct  validity  addresses  the  issue  of  whether  a  survey  is 
measuring  what  it  intends  to  measure.  In  this  section,  operational 
mitigations  used  to  improve  construct  validity  are  described. 

There  are  three  conditions  for  ensuring  valid  responses  to  survey 
questions.  The  respondents  must 

1.  understand  the  question 

2.  be  capable  of  providing  the  information  requested 

3.  be  willing  to  provide  the  information  requested 

While  all  three  are  needed,  take  particular  care  in  wording  each  question 
so  that  the  respondents  understand  it.  The  wrong  choice  of  words  can 
create  any  number  of  problems.  If  the  question  is  ambiguous  or 
confusing,  the  response  will  be  worse  than  valueless — it  will  be 
misleading. 

Questions  must  be  carefully  designed  in  language  that  is  readily 
understandable  by  the  respondents. 


While  no  one  will  argue  about  the  importance  of  choosing  the  right 
words,  providing  satisfactory  guidelines  for  wording  questions  is  not 
without  challenges.  The  problem  is  that  the  rules,  admonitions,  and 
principles  for  how  to  word  questions,  while  generally  good,  often  conflict 
with  one  another.  Consider  this  list  of  question-wording  admonitions  from 
texts  on  questionnaire  design: 

•  Do  not  be  vague. 

•  Use  simple  words. 

•  Keep  it  short. 

•  Be  specific. 

•  Avoid  bias. 

•  Do  not  be  too  specific. 

•  Avoid  hypothetical  questions. 

The  problem  is  that  these  guidelines  often  get  in  one  another’s  way. 

For  example,  “use  simple  words”  is  good  advice,  but  it  frequently 
interferes  with  advice  to  “keep  it  short.”  The  advice,  “do  not  be  vague” 
can  lead  to  questions  that  are  “too  specific.” 
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4.2.2  Question  Wording  (continued) 


Questions  have  a 
context 


Using  the 
audience  analysis 


Some  common 
wording  problems 


The  reason  that  seemingly  good  advice  may  turn  out  to  be  bad  advice  is 
that  questions  are  not  written  in  the  abstract.  Questions  are  written  for 

1 .  a  particular  target  audience  (or  population) 

2.  a  particular  purpose 

3.  placement  next  to  other  questions  in  the  questionnaire 

Words  that  are  too  difficult  for  use  with  some  audiences  might  be 
perfectly  acceptable  for  others.  A  question  that  is  vague  may  satisfy  the 
objectives  of  an  exploratory  study,  but  not  the  objectives  of  an  analytic 
study.  A  question  that  may  not  make  sense  by  itself  may  be  very  clear 
when  it  is  asked  with  ones  that  directly  precede  it. 

This  is  why  the  audience  analysis  (a  product  of  stage  2)  is  such  an 
important  resource  during  this  substage. 


Questions  must  be  worded  for  the  population  of  individuals  characterized 
in  the  audience  analysis.  This  means  that  the  researcher  must  take  into 
consideration  the  education,  vocabulary,  culture,  and  experiences  of 
the  respondents. 


In  his  book  titled  Mail  and  Telephone  Surveys:  The  Total  Design 
Method,  Dillman  recognized  that  simple  admonitions  about  wording, 
no  matter  how  well  intended,  cannot  be  considered  absolute  principles 
that  must  be  adhered  to  without  exception.  Instead,  he  organized  a  list 
of  common  wording  problems  as  a  series  of  questions  that  can  and 
should  be  asked  about  each  item  a  researcher  considers  for  inclusion  in 
a  survey  [Dillman  78]. 

They  are: 

•  Will  the  words  be  uniformly  understood? 

•  Do  the  questions  use  unconventional  terminology,  phrases, 
or  abbreviations? 

•  Are  the  questions  too  cryptic,  vague,  or  precise? 

•  Is  the  question  biased? 

•  Is  the  question  objectionable? 

•  Is  the  question  too  demanding? 

•  Is  it  a  compound  question? 

•  Does  the  question  include  a  double  negative? 

•  Are  the  answer  choices  mutually  exclusive  and  exhaustive? 

•  Have  you  assumed  too  much  about  respondent  knowledge 
or  behavior? 

•  Is  the  question  technically  accurate? 
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4.2.2  Question  Wording  (continued) 


Wording  uniformly 
understood? 


Unconventional 
terminology  and 
abbreviations? 


Vague  or  too 
precise? 


If  you  want  the  question  to  mean  the  same  thing  to  all  respondents, 
keep  the  wording  as  clear  and  unambiguous  as  possible.  Do  not  use 
words  that  are  needlessly  complex.  A  thesaurus  is  an  indispensable 
tool  for  finding  replacement  words  for  those  that  can  be  simplified. 

Piloting  the  survey  with  representatives  of  the  target  audience  is  very 
important  because  it  helps  you  find  the  commonly  shared  vocabulary 
of  the  target  audience.  Piloting  is  discussed  in  stage  5  of  the  survey 
process. 


Be  careful  to  avoid  technical  terminology  that  might  not  be  universally 
understood  by  the  target  audience.  This  is  a  common  mistake  made  by 
subject  matter  experts  fielding  a  questionnaire  for  systems  and 
software  engineering  practitioners,  While  certain  terms  may  be  defined 
a  certain  way  in  the  technical  literature,  in  practice  the  same  words  can 
have  very  different  interpretations  and  meanings.  For  example,  the 
following  terms  are  interpreted  differently  across  the  community: 

•  software  (or  systems)  architecture 

•  integrated  product  team 

•  Six  Sigma 

•  process  capability 

•  requirements  engineering  and/or  management 

Avoid  using  abbreviations  unless  you  are  confident  that  the 
abbreviation  is  universally  understood  by  your  target  audience. 


People  interpret  vague  terminology  in  many  different  ways.  Therefore, 
their  responses  will  be  equally  varied.  Varied  replies  due  to  vague 
wording  render  the  responses  meaningless.  (Don’t  forget  that  in  a  self- 
administered  questionnaire,  there  is  no  way  to  test  understanding  or 
probe  further  with  follow-up  questions.) 

When  evaluating  vagueness  of  wording  and  the  overall  meaning  of 
questions,  it  is  important  not  to  isolate  the  questions  from  others  in  the 
questionnaire.  In  a  series  of  questions,  a  term  that  would  otherwise  be 
vague  might  be  absolutely  clear  in  the  context  of  the  other 
accompanying  questions. 

On  the  other  hand,  the  desire  to  avoid  vagueness  can  sometimes  result 
in  a  question  being  so  precise  that  respondents  have  difficulty 
answering  it.  For  example,  it  may  be  too  much  to  ask  respondents  how 
many  times  they  performed  an  activity  during  the  last  year.  A  better 
way  to  ask  the  question  is  to  list  number  ranges  from  which 
respondents  can  choose  without  having  to  be  overly  precise. 
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4.2.2  Question  Wording  (continued) 


Is  the  question  Bias  must  be  avoided  in  questions.  This  means  that  questions  should 
biased?  phrased  carefully  to  avoid  suggesting  that  certain  answers  are  more 

acceptable  than  others. 

Moser  and  Kalton  have  defined  “bias”  as  occurring  if  the  content, 
structure,  or  wording  of  a  question  leads  respondents  in  the  direction 
of  a  certain  answer.  This  can  occur  with  leading  questions  that  include 
a  component  of  expectation  within  them  [Moser  89]. 

In  his  excellent  primer,  The  Art  of  Asking  Questions  b  Payne  illustrated 
,  the  influence  of  leading  questions  in  a  story  of  a  snack-bar  waiter  who 

increased  spectacularly  the  sale  of  eggs  in  milk  shakes.  He  asked 
customers  not  whether  they  wanted  an  egg  or  not,  but  whether,  in  fact, 
they  wanted  one  or  two  (the  choice  of  zero  eggs  was  not  included) 
[Payne  80] .  While  this  is  a  fine  sales  technique,  it  would,  of  course,  be 
unsuitable  for  scientific  research. 


Is  the  question  Respondents  might  find  a  question  objectionable  for  a  number  of 
objectionable?  reasons.  For  example, 

•  they  might  consider  the  question  too  personal  and  feel  that  no  one 
has  the  right  to  that  information 

•  they  might  find  the  question  incriminating  if  they  are  being  asked 
to  expose  behavior  that  violates  their  organization’s  policies.8 

The  researcher  needs  to  be  careful  since  it  only  takes  a  few 
objectionable  questions  to  make  the  respondent  refuse  to  answer  any 
part  of  the  questionnaire.  The  researcher  must  always  ensure  that 
information  will  not  be  disclosed  in  a  way  that  allows  it  be  attributed 
to  any  single  respondent. 

In  some  cases,  technical  researchers  will  propose  questions  that  require  a 
great  deal  of  mental  processing  on  the  part  of  the  respondent.  In  these 
cases,  the  patience  of  even  the  most  enthusiastic  survey  respondent  may 
be  challenged.  The  lack  of  an  interviewer  who  can  explain  how  certain 
questions  are  to  be  answered  makes  it  essential  that  questions  sometimes 
used  in  face-to-face  interviews  be  modified  or  eliminated  from  self- 
administered  questionnaires. 

There  are  two  issues  here:  (1)  telescoping,  and  (2)  forgetting.  For 
many,  memory  has  a  telescoping  effect,  where  two  months  seem  like 
one,  and  a  year  ago  seems  like  six  months.  It’s  unreasonable  to  think 
that  individuals  can  call  up  detailed  accounts  of  behaviors  that  have 
occurred  a  week  ago,  let  alone  during  the  last  year. 

_  Continued  on  next  page 

8  Even  though  the  source  of  survey  responses  are  anonymous,  many  people  may  not  want  to  take 
the  risk  of  being  identified  as  the  author  of  a  given  response. 


Is  the  question 
too  demanding? 


CMU/SEI-2005-HB-004 


51 


4.2.2  Question  Wording  (continued) 


Is  the  question 
too  demanding? 
(continued) 


Although  there  is  no  doubt  that  accurate  answers  to  the  questions 
would  be  of  great  value,  it  is  doubtful  whether  people  are  willing  or 
capable  of  providing  them — especially  in  cases  where  the  respondent 
is  a  busy  individual  with  many  responsibilities.  These  types  of 
questions  must  be  avoided. 


Are  you  using 
a  compound 
question? 


Compound  questions  are  sometimes  referred  to  as  double-barreled 
questions.  They  are  separate  questions  that  have  been  combined  into 
one,  yet  the  respondent  is  asked  to  provide  a  single  answer.  In  these 
cases,  you  cannot  be  sure  which  part  of  the  question  the  respondent  is 
really  answering  and  the  response  cannot  be  interpreted  correctly. 

Never  ask  two  questions  in  one  questionnaire  item. 


Double¬ 

negatives? 


Never  use  double-negatives  in  survey  questions.  These  typically  occur 
when  yes/no  or  agree/disagree  are  provided  as  response  options. 

Here  is  an  example  of  a  double  negative: 

Should  the  project  manager  not  be  responsible  for  quality 
standards  within  his  team? 

I~~l  Yes 
□  No 


Here  is  a  suggested  revision  to  avoid  the  double-negative: 


In  your  organization,  who  is  responsible  for  quality  standards? 


□ 

□ 

□ 

□ 


Program  manager 
Project  manager 
Staff  members 
Don’t  know 
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4.2.2  Question  Wording  (Continued) 


Choices  mutually 
exclusive  and 
exhaustive? 


Have  you 
assumed  too 
much  about  the 
respondent? 


Is  the  question 

technically 

accurate? 


Section  summary 


Be  careful  to  avoid  providing  response  options  that  are  not  mutually 
exclusive.  For  example,  consider  the  following  question  and  response 
set. 

How  many  years  of  systems  engineering  experience  do  you  have? 

□  0-3 

■  ;r  □  3-5  :>;f>  • 

□  5-10 

n  >io 

Note  that  individuals  who  have  three  years  experience  could  choose 
the  first  or  second  response.  Those  with  five  years  of  experience  could 
choose  the  second  or  third  response. 


In  almost  all  questions,  the  researcher  assumes  the  respondent 
possesses  a  certain  amount  of  knowledge.  However,  the  problem  of 
assuming  too  much  knowledge  in  surveys  is  prevalent — especially  in 
technical  domains. 

To  avoid  this  problem,  always  consider  including  an  option  such  as 
“Don’t  Know”  as  part  of  the  response  set.  Another  way  to  avoid  the 
problem  is  to  include  a  response  option  such  as  “It  depends,”  coupled 
with  a  prompt  to  the  responder  to  explain  their  selection  of  that  option. 

False  assumptions  might  also  be  made  about  respondent  behavior. 
Consider  including  the  response  “None  of  the  above”  as  part  of  the 
response  set.  Or  include  a  partially  closed-ended  response  category 
that  includes  “Other”  and  have  respondents  specify  their  definition  of 
“Other.” 


In  some  cases,  information  is  provided  as  part  of  the  question. 
Therefore,  it  is  important  that  this  information  is  technically  accurate, 
leaving  no  doubt  of  the  researcher’s  understanding  of  the  topic  being 
investigated. 


In  this  section,  we  have  reviewed  a  number  of  points  that  the 
researcher  should  ask  of  every  question  in  their  survey.  If  there  is  a 
negative  answer  to  any  of  these  questions,  then  the  question-writing 
task  is  incomplete. 
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4.3  Design  Question  Sequence  and  Overall 
Questionnaire  Layout 


—  4.1  Determine  the  questions  to  be  asked 

—  4.2  Select  the  question  type/format  tor  each  question  and 

specify  wording 

—  4.3  Design  the  question  sequence  and  overall  questionnaire  layout 

—  4.4  Develop  ancillary  documents 


Introduction  At  this  point  in  the  survey  development  process,  you  should  have  a  list 

of  questions  that  you  believe  address  the  research  objectives  of  the 
survey.  But  there  is  still  more  work  to  do  before  your  questionnaire  is 
complete  and  ready  to  be  tested  and  distributed  to  your  target 
audience. 

To  complete  the  survey  questionnaire,  you  will  need  to  consider  the 

•  length  of  the  questionnaire 

•  organization  and  sequencing  of  questions 

•  instructions  and  questionnaire  section  introductions 

•  page  layout  options,  including 

-  hardcopy  questionnaires  for  mailing 

-  Web-hosted  questionnaire 

-  email 

-  other,  such  as  commercial  survey  software 


In  this 
section 


Section 

See  page 

Determining  the  length  of  your  questionnaire 

55 

Organizing  and  sequencing  your  questions 

57 

Instructions  and  transition  paragraphs 

58 

Questionnaire  page  layout  considerations — paper  form 

60 

Web-based  questionnaires 

63 
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4.3.1  Determining  the  Length  of  Your  Questionnaire 


Introduction 


Has  the  size 
of  your 
questionnaire 
grown? 


How  long 
can  your 

questionnaire  be? 


Many  researchers  make  the  mistake  of  asking  too  many  questions. 

This  often  arises  from  an  incomplete  analysis  of  how  to  meet  the 
survey  objectives.  Questionnaire  length  is  one  of  the  most  frequent 
reasons  that  potential  survey  respondents  give  when  refusing  to 
participate  in  a  survey  [Burchell  92].  Your  greatest  enemy  in  survey 
research  may  well  be  poor  response  rate. 

Determining  the  appropriate 
length  of  your  questionnaire  is 
like  walking  a  tightrope.  On 
the  one  hand,  you  want  to 
make  the  most  of  this 
opportunity  to  ask  your  target 
audience  questions.  On  the 
other  hand,  if  your 
questionnaire  is  too  long, 
potential  respondents  will  view 
it  as  something  they  do  not  have  the  time  to  complete  and  will  not 
participate.  If  you  have  a  poor  response  rate,  then  your  research  study 
will  be  weak. 

As  the  primary  researcher,  you  want  to  do  everything  within  your 
power  to  promote  a  high  response  rate  from  your  audience.  The  longer 
the  questionnaire,  the  less  likely  that  individuals  will  be  willing  to 
participate  in  it. 


#  of  willing  participants 


In  the  paragraph  labeled  “Why  prioritize  questions?”  on  page  39,  you 
are  encouraged  to  begin  considering  size  constraints  for  the 
questionnaire.  Now,  as  a  result  of  transforming  internal  questions  into 
suitable  questions  for  respondents  (during  substage  4.2),  you  may  find 
that  the  number  of  questions  has  increased  dramatically — perhaps  you 
now  have  double  or  even  triple  the  number  of  internal  questions  you 
had  at  the  start.  This  occurs  because  a  single  internal  open-ended 
question  can  require  several  closed-ended  questions  to  address  the 
intent  of  the  original  internal  question. 

You  must  now  take  a  hard  look  at  what  you  have  and  make  difficult 
decisions  to  pare  down  the  number  of  questions  in  your  survey. 


Numerous  studies  have  shown  the  significant  effect  of  questionnaire 
length  on  response  rates.  Each  additional  question  can  reduce  response 
rate  by  0.5%,  and  each  additional  page  by  5%  [Heberlein  78]. 

There  is  a  significant  decrease  in  response  rates  for  questionnaires 
longer  than  four  pages  [Yammarino  91]. 


CMU/SEI-2005-HB-004 


55 


4.3.1  Determining  the  Length  of  Your  Questionnaire 


Effect  on  Questionnaire  length  is  also  believed  to  affect  data  quality.  As  the 

data  quality  time-to-complete  duration  lengthens,  respondents  can  become  tired, 

annoyed,  bored,  or  distracted  by  external  factors.  This  can  decrease  the 
degree  of  effort  and  thought  that  respondents  invest  in  answering 
questions  [James  90].  Other  findings  on  the  effect  of  questionnaire 
length  on  data  quality  generally  suggest  overly  long  questionnaires 
have  a  negative  influence  on  data  quality  [Burchell  92,  Helgeson  94, 
Herzog  81]. 


Exceptions  to  If  you  believe  your  questionnaire  is  bordering  on  being  too  lengthy, 
the  rule?  there  are  some  tactics  you  can  employ  to  promote  participation. 

Perception  of  length  can  be  moderated  by  other  characteristics  of  the 
survey  or  survey  process.  For  example,  if  respondents  are  interested  in 
the  topic  of  the  survey  they  might  be  willing  to  go  the  extra  mile  to 
complete  a  lengthy  questionnaire.  Promising  to  inform  the  respondents 
about  the  survey’s  results  can  sometimes  improve  the  response  rate.  Other 
incentives  (e.g.,  financial,  prizes,  appealing  to  philanthropy)  can  also 
mitigate  the  negative  impact  of  questionnaire  length  on  response  rate. 

However,  keep  in  mind  that  incentives  are  typically  employed  to  boost 
participation  even  when  the  questionnaire  is  a  reasonable  length. 


How  to  decide 
which  questions 
to  include 


If  your  questionnaire  is  too  long  and  you  need  to  reduce  the  length, 
review  each  of  your  questions  carefully  with  respect  to  the  objectives 
that  were  developed  during  stage  1  of  this  process.  For  each  question, 
ask  yourself  what  you  will  do  with  the  information.  If  you  cannot  give 
yourself  a  satisfactory  answer,  leave  it  out.  Avoid  the  temptation  to  add 
a  few  more  questions  just  because  you  are  doing  a  survey  anyway.  It 
may  help  to  place  your  questions  into  three  groups: 

1 .  must  know 

2.  useful  to  know 

3.  nice  to  know 

Discard  group  3  unless  groups  1  and  2  are  very  short. 

The  best  way  to  categorize  your  questions  is  to  invite  your  subject 
matter  experts  and  the  management  that  sponsored  the  survey 
development  project  to  participate.  You  can  use  a  structured  group 
decision-making  tool  such  as  multi-voting9  to  select  the  highest 
priority  questions  for  your  survey. 


9  Scholtes  describes  the  multi-voting  technique  [Scholtes  88], 
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4.3.2  Organizing  and  Sequencing  Your  Questions 


Introduction 


Question 

categories 


Grouping 
and  organizing 
questions 


Sequencing  your 
questions 


This  section  describes  the  way  questions  in  your  survey  questionnaire 
should  be  arranged  and  organized.  Sections  of  your  questionnaire  also 
will  need  to  be  introduced  appropriately  with  explanatory  information 
and  clear  and  concise  instructions  for  the  respondent. 


The  table  below  describes  the  three  basic  categories  in  a  questionnaire. 


Category 

Description 

Demographic 

These  are  questions  about  respondents’  characteristics  and 
circumstances.  They  are  often  used  to  compare  answers  to 
substantive  questions  between  respondents  with  different 
characteristics  (e.g.,  managers  vs.  staff,  men  vs.  women, 
experienced  vs.  inexperienced). 

Substantive 

These  are  questions  that  are  the  substance  of  the  survey — the 
ones  that  address  the  survey  objectives. 

Filter 

In  some  surveys,  there  are  questions  that  do  not  apply  to 
all  respondents.  Any  question  whose  response  determines 
whether  another  question  is  to  be  responded  to  is  a 
filter  question. 

Don’t  skip  around  from  topic  to  topic.  Just  as  in  writing  a  paper, 
prepare  an  outline  and  group  your  questions  according  to  topic  or 
subject  matter.  Write  transitional  statements  between  sections  or 
changes  in  subject  matter. 

Demographic  questions  are  fact-based  questions,  as  opposed  to 
substantive  questions,  which  are  typically  opinion  based.  Demographic 
questions,  which  are  typically  easier  to  answer,  are  usually  asked  at  the 
beginning  of  the  questionnaire.  Asking  the  demographic  questions  gets 
respondents  engaged  in  the  questionnaire  and  warmed  up  before 
answering  more  challenging  and  reflective  questions  about  their 
opinions. 

The  substantive  questions  should  follow.  If  your  substantive  questions 
include  any  open-ended  questions,  these  should  be  positioned  towards 
the  end  of  the  questionnaire  since  it  is  usually  easier  to  respond  to 
closed-ended  questions. 


Providing  the  proper  order  is  not  a  trivial  issue.  The  location  of  a 
question  must  be  considered  in  light  of  how  it  will  affect  questions  that 
follow  it.  Questions  should  flow  in  a  logical  order. 
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4.3.3  Instructions  and  Transition  Paragraphs 


Introduction 


Non-question  text 


Survey  title 


Writing 

instructions 


Thus  far,  you  have  grouped  the  questions  by  subject  matter  and 
carefully  sequenced  them.  Of  course,  this  was  not  a  one-shot  effort  and 
probably  required  several  iterations  before  the  organization  was 
correct. 

In  this  section  the  importance  of  non-question  information  critical  to 
obtaining  quality  information  is  discussed. 


In  addition  to  the  questions,  you  will  need  to  include  other  non¬ 
question  information  in  your  questionnaire,  including 

•  survey  title 

•  instructions 

•  transition  paragraphs 

•  definitions 


A  descriptive  survey  title  should  appear  at  the  top  of  the  questionnaire. 
You  should  also  consider  including  the  title  as  part  of  the  header  or 
footer  of  each  page  of  the  questionnaire  (if  it  is  a  paper  or  emailed 
form)  or  at  the  top  or  bottom  of  the  screen  (if  it  is  a  Web-based  form). 


For  self-administered  questionnaires,  you  must  ensure  that  the 
respondents  understand  how  to  answer  each  question.  Unlike  face-to- 
face  interview  surveys  or  telephone  surveys,  self-administered  surveys 
are  self-contained  and  respondents  do  not  have  the  opportunity  to  ask 
questions  about  how  to  complete  the  survey  questionnaire. 

The  method  for  answering  each  question  must  be  spelled  out.  Tell 
respondents  exactly  how  to  answer.  For  example, 

•  Does  the  question  require  a  single  answer  only,  or  should  they 
select  all  answers  that  apply,  or  should  they  select  a  specific 
number  of  alternatives  that  have  been  provided  by  the 
questionnaire  item? 

•  Should  a  question  be  answered  by  checking  a  box?  Or  by  choosing 
from  a  pull-down  menu?  Or  by  writing  a  brief  answer? 

Instructions  must  be  worded  clearly  and  appear  next  to  the  questions 
involved. 

Never  assume  that  respondents  will  understand  what  they  should  do 
unless  you  specifically  instruct  them  how  to  do  it. 
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4.3.3  Instructions  and  Transition  Paragraphs  (continued) 


Writing  transition 
paragraphs 


Writing 

definitions 


If  your  questionnaire  includes  multiple  sections,  provide  brief 
transitional  paragraphs  to  orient  respondents  to  what  is  coming  up. 

An  example  of  a  transitional  paragraph  that  prepares  a  respondent  for 
answering  a  group  of  questionnaire  items  is  presented  below. 

Example 

The  following  items  ask  about  the  types  of  training  you  have  received 
in  relation  to  your  existing  position  and  responsibilities. 


You  should  always  try  to  use  words  that  are  easily  understood  by  your 
audience  and  do  not  require  definition.  However,  in  the  domains  of 
systems  and  software  engineering,  there  are  many  terms  that  defy 
universal  definition.  For  these  cases,  you  must  let  the  respondent  know 
how  to  interpret  them. 

There  are  various  ways  to  accomplish  this.  If  there  are  few  terms  in 
your  questionnaire  that  require  definitions,  it  is  fine  to  provide  the 
definition  as  part  of  the  question.  But,  if  there  are  many  terms  you 
suspect  would  not  be  uniformly  interpreted,  you  might  want  to  provide 
a  glossary  section  as  part  of  your  questionnaire.  Each  term  in  the 
glossary  could  be  italicized  or  underlined  within  a  question  to  indicate 
that  it  appears  in  the  glossary. 

If  you  are  mediating  your  questionnaire  using  the  Web,  then  consider 
using  hypertext  so  that  the  respondent  can  click  on  a  term  to 
immediately  see  the  operational  definition  as  you  intend  it. 
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4.3.4  Questionnaire  Page  Layout  Considerations — Paper 
Form 

Introduction  This  section  provides  page  layout  guidance  for  questionnaires  mailed 

to  respondents  in  paper  form  or  sent  as  attachments  in  email. 

The  design  and  print  quality  of  your  questionnaire  will  convey  a 
1  certain  meaning  to  respondents.  Forms  that  look  attractive  impress 
upon  the  recipient  that  care  has  been  taken  in  their  development,  and 
an  effective  page  layout  also  makes  your  questionnaire  easier  to 
understand.  Your  questionnaire  should  be  easy  to  read  and  complete. 


What  if  you  are 
planning  to  use 
a  Web-based 
questionnaire? 


Although  you  may  be  planning  to  develop  a  Web-based  version  of 
your  questionnaire  for  respondents  to  complete,  you  should  also 
prepare  a  paper  form  for  those  individuals  who  have  difficulty  using 
the  Internet  form. 

Due  to  differences  in  Web  browser  types  and  versions,  some  of  your 
intended  respondents  may  request  a  paper  copy,  and  you  need  to  be 
prepared  by  having  one  available. 


“Right  and  wrong  do  not  exist  in  graphic  design.  There  is  only 

effective  and  non-effective  communication”  [Bilak  95]. 

Use  the  following  heuristics  to  guide  the  layout  of  your  questionnaire: 

•  Keep  your  answer  choices  in  a  straight,  neat  column.  (See  the 
example  after  this  list  of  guidelines.) 

•  Nothing  improves  a  bad  layout  faster  than  a  good  dose  of 
nothingingness.  White  space  is  not  wasted  space.  It  provides  visual 
breathing  room  for  the  eye.  Add  white  space  to  make  a  page  less 
cramped,  confusing,  or  overwhelming.  Do  not  cram  too  many 
questions  on  a  single  page.  Proper  spacing  makes  it  easier  to  read 
and  process  the  information.  (See  the  example  after  this  list  of 
guidelines.) 

•  Do  not  split  a  question  across  two  pages.  Each  question,  and  its 
possible  responses,  if  provided,  should  appear  on  the  same  page.  If 
the  entire  question  does  not  fit  on  a  page,  then  change  the  order  of 
questions  (if  possible),  or  simply  leave  a  blank  area  at  the  bottom 
of  the  page. 

•  Use  the  right-hand  edge  of  the  paper  for  answer  choices.  (Eye 
tracking  studies  show  that  this  is  the  best  place  to  position  the 
answer  choices.)  Using  the  right  edge  is  also  easiest  for  data  entry 

•  Do  not  use  too  many  different  types  of  fonts,  font  sizes,  or  font 
colors.  Doing  so  is  distracting  to  the  reader. 

Continued  on  next  page 
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4.3.4  Questionnaire  Page  Layout  Considerations — Paper  Form  (continued) 


Guidelines 

(continued) 


•  Use  bold-face,  italics,  and  different  font  colors  sparingly.  Develop 
conventions  for  using  font  changes  in  your  questionnaire  and  be 
consistent. 

•  Graphics  should  be  used  only  if  they  serve  a  purpose. 

•  Do  not  cross-reference  information  in  a  different  part  of  the 
questionnaire.  Place  information  the  respondent  needs  where  they 
need  it. 

•  To  help  individuals  with  poor  eyesight: 

-  Always  use  a  font  size  of  at  least  10  points. 

-  Choose  strongly  contrasting  paper  and  font  color. 

•  Never  use  ALL  CAPS.  Typing  in  ALL  CAPS  is  considered 
shouting  and  is  frowned  on  in  most  cases. 

•  Include  page  numbers. 

•  Include  the  survey  title  as  part  of  the  header  or  footer  on  all  pages 
except  the  title  page. 


Example:  The  example  below  contrasts  poor  alignment  with  the  appropriate 

alignment  alignment  of  answer  choices. 

of  answer  choices 

Bad  -  Poor  alignment  of  answer  choices: 

j  The  acquisition  project  team  ensures  there  is  no  loss  in  continuity  of  support  to  the  software 

products  during  transition  from  the  development  contractor  to  the  software  support  organization. 
[H  Agree  ED  Unsure  HU  Disagree 

2  Configuration  management  of  the  software  products  is  maintained  throughout  the  transition. .  HI 
Agree  HI  Unsure  HI  Disagree 

3  The  strategy  for  transition  into  maintenance  is  documented,  communicated,  and  agreed  to  by  all 
parties  early  in  the  acquisition.  HI  Agree  HI  Unsure  HI  Disagree 


Good  -  Appropriate  alignment  of  answer  choices: 

1  The  acquisition  project  team  ensures  there  is  no  loss  in  continuity  of 
support  to  the  software  products  during  transition  from  the  development 
contractor  to  the  software  support  organization. 

2  Configuration  management  of  the  software  products  is  maintained 
throughout  the  transition. 

3  The  strategy  for  transition  into  maintenance  is  documented, 
communicated,  and  agreed  to  by  all  parties  early  in  the  acquisition. 


Agree  Unsure  Disagree 

□  □  □ 

□  □  □ 

□  □  □ 
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4.3.4  Questionnaire  Page  Layout  Considerations — Paper  Form  (continued) 


Example:  use  The  example  below  contrasts  the  lack  of  appropriate  white  space  to  the 

of  white  space  effective  use  of  white  space.  Note  that  appropriate  white  space  makes 
the  questions  easier  to  read  and  respond  to. 

Agree  Unsure  Disagree 

□  □  □ 

□  □  □ 

□  □  □ 

Good  -  Appropriate  use  of  white  space:  Agree  Unsure  Disagree 

j  The  acquisition  project  team  ensures  there  is  no  loss  in  continuity  of  support  to  '  CD  ED  ED 

the  software  products  during  transition  from  the  development  contractor  to  the 
software  support  organization. 

2  Configuration  management  of  the  software  products  is  maintained  throughout  ED  ED  ED 

the  transition. 

2  The  strategy  for  transition  into  maintenance  is  documented,  communicated,  and 
agreed  to  by  all  parties  early  in  the  acquisition. 


□  □  □ 


1 .  The  acquisition  project  team  ensures  there  is  no  loss  in  continuity  of  support  to 
the  software  products  during  transition  from  the  development  contractor  to  the 
software  support  organization. 

2  Configuration  management  of  the  software  products  is  maintained  throughout 
the  transition. 

3  The  strategy  for  transition  into  maintenance  is  documented,  communicated,  and 
agreed  to  by  all  parties  early  in  the  acquisition. 
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4.3.5  Web-Based  Questionnaires 


Introduction 


Skills  needed 


Availability  of 
Web-based 
survey  products 


Impact  of  large- 
volume  traffic  on 
your  system 


There  has  been  a  strong  upward  trend  in  the  use  of  Internet-based 
surveys.  Surveys  mediated  through  the  World  Wide  Web  are  popular 
for  a  number  of  reasons  that  will  be  discussed  in  this  section. 


The  guidelines  for  developing  paper-based  questionnaires  also  apply  to 
Web-based  questionnaires.  However,  there  are  additional  issues  that 
you  must  consider  when  implementing  Web-based  questionnaires. 

Unlike  paper-based  questionnaires,  the  design  and  development  of 
Web-based  questionnaires  require  Web  page  design  skills  and 
computer  programming  expertise. 


There  are  many  companies  that  provide  products  and  services  for 
implementing  Web-based  surveys.  A  list  of  these  companies  can  be 
obtained  by  searching  the  Web. 

This  section  provides  some  high-level  guidance  for  the  development  of 
a  Web-based  survey  application.  You  might  also  want  to  consider 
attending  training  if  you  are  planning  to  develop  your  own.  Service- 
providers  that  offer  training  can  be  located  through  a  Web  search. 

This  section  does  not  attempt  to  address  computer  language  issues 
involved  in  developing  a  Web-based  survey  application.  However, 
various  design  criteria  and  issues  are  discussed  that  can  be  used  to 
evaluate  alternative  service  providers. 


If  you  plan  to  conduct  an  extensive  survey  with  a  large  sample,  you 
need  to  understand  the  network  performance  impact  of  having  large 
numbers  of  respondents  simultaneously  log  on  to  the  server  hosting 
the  Web-based  questionnaire. 

You  will  need  to  work  with  your  network  system  administrator  to 
ensure  that  the  system  can  withstand  the  volume  of  traffic  that  can 
result  from  simultaneous  access  to  the  Web-based  questionnaire.  The 
impact  on  network  connectivity  and  on  database  transaction 
performance  should  be  understood  before  a  Web-based 
implementation  is  chosen  for  mediating  your  survey  research. 
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4.3.5  Web-Based  Questionnaires  (continued) 


Advantages  of 

Web-based 

surveys 


Providing  access 
to  the 

questionnaire 


Screen 

presentation 

styles 


Web-based  surveys  possess  a  number  of  advantages  over  mailed  paper 
surveys.  These  include 

•  a  faster  response  rate 

•  the  ease  of  sending  reminders  to  survey  participants 

•  the  ease  of  processing  data 

•  dynamic  error-checking  capability 

•  the  ability  to  make  complex  skip  pattern  questions  easier  to  follow 
(see  previous  mention  of  filter  questions  on  page  57) 

•  the  use  of  pop-up  definitions  and  instructions  for  selected  questions 

•  the  use  of  drop-down  boxes 


To  protect  the  validity  of  a  Web-based  survey,  access  must  be  carefully 
controlled.  This  is  done  to 

•  prevent  unauthorized  individuals  from  participating  in  the  survey 

•  prevent  duplicate  submissions  by  the  same  respondent 

•  ensure  that  respondents’  privacy,  and  their  perception  of  privacy,  is 
protected 

Access  can  be  restricted  by  emailing  candidate  respondents  a  unique 
personal  identification  number  (PIN)  or  a  usemame/password 
combination  to  access  the  Web-based  survey.  (“Candidate 
respondents”  are  those  who  have  been  selected  as  a  result  of  the 
sampling  plan  in  stage  3  of  this  process.) 


Unlike  paper-based  questionnaires,  Web-based  questionnaires  appear 
on  a  computer  screen  and  should  be  designed  accordingly. 

There  are  two  primary  methods  for  presenting  information  to  the 
respondent: 

•  questionnaires  that  provide  questions  screen-by-screen 

•  questionnaires  that  allow  scrolling 

With  the  screen-by-screen  method,  respondent  must  complete  a 
question  (or  a  few  questions)  before  proceeding  to  the  next  screen  of 
questions.  Scrollable  questionnaires  allow  vertical  scrolling,  giving  the 
respondent  the  entire  view  of  the  questionnaire  or  a  section  of  the 
questionnaire. 

Of  course,  some  combination  of  the  two  methods  can  be  implemented, 
if  appropriate.  Also,  questionnaires  can  differ  greatly  within  these 
categories  because  of  variations  in  layout  and  methods  of  navigation. 

Research  has  concluded  that  a  generic  design  format  for  all  Web-based 
questionnaires  is  not  as  suitable  as  designing  each  questionnaire  to 
reflect  its  purpose.  Some  designs  are  more  suitable  for  certain 
audiences,  purposes,  or  types  of  questions  than  others  [Couper  00], 
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4.3.5  Web-Based  Questionnaires  (continued) 


Graphic  design 
issues 


The  welcome 
screen 


Navigation 


Choice  of 
response  format 


Web-based  design  may  require  more  thought  when  it  comes  to  color 
schemes,  font  choices,  and  layout  options.  Take  advantage  of  the 
wealth  of  information  available  in  graphic  design  theory.  Poor  choice 
of  color  design  or  small  or  inconsistent  fonts  can  lead  to  respondents 
aborting  the  survey. 

If  you  belong  to  an  organization  that  maintains  a  graphic  design  staff, 
then  by  all  means  obtain  their  support  and  feedback. 


Begin  your  questionnaire  with  a  welcome  screen  that  is  clear,  concise, 
and  provides  general  instructions.  This  screen  should  display  many  of 
the  information  items  included  in  cover  letters  used  for  paper-based 
questionnaires.  (See  page  72  for  content  guidelines  for  a  survey  cover 
letter.) 


If  your  questionnaire  contains  multiple  sections,  you  need  to  clearly 
indicate  that  additional  items  must  be  accessed  either  by  scrolling  or 
by  clicking  on  a  control  button  to  move  to  the  next  screen. 

Respondents  should  be  able  to  navigate  easily  through  the 
questionnaire — streamlining  forward  through  the  questionnaire  and 
backtracking  so  answers  can  be  changed  (unless  the  effects  of  the 
question  order  are  a  concern  of  the  research  team). 


There  are  a  variety  of  response  formats  that  are  available  with  Web- 
based  surveys,  including 

•  radio  buttons 

•  check  boxes 

•  drop-down  boxes 

•  text  input 

•  text  area 

These  are  explained  in  further  detail  in  upcoming  paragraphs. 
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4.3.5  Web-Based  Questionnaires  (continued) 


Radio  buttons 


Check  boxes 


Drop-down  boxes 


Radio  buttons  are  appropriate  for  questions  that  require  a  “select  only 
one”  choice  from  mutually  exclusive  items. 

Example:  Radio  buttons 

Q10.  What  title  best  describes  your  position  in  your  organization? 

O  Program  Director 
O  Project  Manager 
O  Engineer 
O  Clerical 
O  Other 


Check  boxes  are  appropriate  for  “check  all  that  apply”  questions. 

Example:  Check  boxes 

Q5.  Which  of  the  following  training  courses  did  you  take  during  the  year  2004? 
Check  all  that  apply. 

□  Project  Management  Basics 

□  Communication  Skills 

□  Performance  Measurement 

□  Intro  to  Negotiation 

□  Acquisition  Management 

Drop-down  boxes  are  used  to  select  one  item  from  a  list  of  options. 

For  long  lists,  allow  type-ahead  look-up  since  it  prevents  tedious 
scrolling. 

Provide  a  visual  cue  for  the  task  in  the  drop-down  box.  For  example, 

“Please  select  one.” 

Example:  Drop-down  boxes 

Q3.  What  state  is  your  company  located  in? 


Select  one 

Alabama 

Alaska 

A  ri7nno 
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4.3.5  Web-Based  Questionnaires  (continued) 


Text  boxes 


Text  areas 


The  text  box  allows  respondents  to  type  in  an  answer  using  their  own 
words.  Text  boxes  are  used 

•  when  a  very  brief  response  to  an  open-ended  question  is  presented 

•  as  part  of  a  partially  closed  question  when  the  provided  response 
options  are  not  appropriate  for  the  respondent.  If  “other”  is  chosen, 
respondents  can  type  in  their  own  responses. 

When  using  text  boxes,  ensure  that  the  size  of  the  box  will 
accommodate  the  amount  of  text  that  is  typed.  Also,  provide  clear 
instructions  for  input  guidance.  For  example,  if  a  calendar  date  is 
required,  provide  guidance  for  the  required  format,  such  as 
MM/DD/YYYY. 

Example:  Text  box 

Q10.  Please  indicate  the  role  that  best  describes  your  position.  If 


you  select  “Other”  type  your  own  position  description. 


A  text  area  allows  a  response  to  open-ended  questions.  For  reasons 
listed  on  page  40,  open-ended  questions  should  be  used  sparingly. 

Text  areas  are  scrollable,  allowing  the  respondent  to  reply  with  as 
many  words  as  the  questionnaire  designer  allows. 

Example:  Text  area 

Q25.  Overall,  what  are  the  one  or  two  most  difficult  problems  that 
your  project  has  faced  in  conducting  successful  acquisition  of 
software-intensive  systems?  (Please  describe  in  your  own 
words.) 
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4.3.5  Web-Based  Questionnaires  (continued) 


Progress  indicator  Research  has  shown  that  respondents  like  to  see  a  progress  indicator  so 
they  can  tell  how  many  questions  remain  in  the  questionnaire.  [Lozar 
02].  Couper  reported  that  a  progress  indicator  may  motivate 
respondents  to  complete  a  questionnaire.  However,  this  positive  effect 
could  be  counteracted  if  the  download  time  increases  due  to  the 
addition  of  a  graphic  process  indicator.  [Couper  01] 


Example:  An  example  of  a  progress  indicator  adapted  from  Couper  is  shown  below 

progress  indicator  [Couper  01]. 


Q5.  Which  of  the  following  training  courses  did  you 
take  during  the  year  2004?  Check  all  that  apply. 

□  Project  Management  Basics  □  Introduction  to  Negotiation 

□  Communication  Skills  □  Acquisition  Management 

□  Performance  Measurement  □  Designing  Web  Pages 


|  Progress 


1:  .1  35  I  %  Done 


V;--. 


Other  design  As  with  paper-based  questionnaires,  page  layout  is  important  and  can 

characteristics  have  a  positive  or  negative  effect  on  respondent  participation. 

The  lack  of  appropriate  spacing  (that  is,  empty  space),  crowded 
horizontal  design,  and  unconventional  layouts  can  make  it  difficult  for 
respondents  to  process  information  and  to  navigate  through  the 
questionnaire.10  Navigation  and  flow  are  important  in  any 
questionnaire,  but  they  are  particularly  important  in  Web-based 
questionnaires  [Redline  99],  Dillman  cites  research  showing  that 
brightness,  fonts,  and  spacing  can  greatly  assist  respondents  in 
navigating  questionnaires,  and  that  conventional  vertical  layout  with 
numbered  questions  and  distinct  and  consistent  space  between 
questions  and  responses  make  Web  questionnaires  respondent-friendly 
[Dillman  01], 


10 


See  paragraph  labeled  “Guidelines”  on  page  9  for  a  discussion  of  the  importance  of  blank  space  in 
page  layout  design. 
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4.3.5  Web-Based  Questionnaires  (continued) 


Allowing 

interruptions 


Error  checking 


Computer  & 

network-based 

issues 


If  your  questionnaire  is  long,  you  should  provide  a  mechanism  that 
allows  respondents  to  interrupt  their  work,  save  it,  and  then  reenter  the 
questionnaire  at  a  later  time.  (You  may  then  want  to  send  a  reminder  to 
the  respondent  that  they  have  not  yet  completed  the  questionnaire.) 

If  you  cannot  provide  the  capability  to  interrupt,  save,  and  resume,  you 
must  make  it  very  clear  to  respondents  that  they  must  complete  the 
questionnaire  in  a  single  sitting.  Still,  in  spite  of  such  a  warning,  you 
can  expect  incomplete  questionnaires. 


One  advantage  of  a  Web-based  questionnaire  is  the  ability  to  provide 
error-checking  capabilities.  Providing  such  a  capability  is  especially 
important  if  your  respondent  population  possesses  varying  degrees  of 
Web  browsing  experience.  In  general,  data  quality  is  improved  with 
error-checking. 

Ideally,  error  messages  should  be  as  specific  as  possible  and  should  be 
placed  directly  above  or  below  the  item.  When  you  provide  error¬ 
checking,  phrase  the  term  “error”  in  a  positive,  helpful  way  and  give 
suggestions  as  to  how  the  question  should  be  completed. 


Keep  in  mind  that  while  there  are  many  advantages  to  implementing 
Web  questionnaires,  there  are  several  issues,  listed  below,  that  could 
cause  some  risk  if  not  addressed  appropriately. 

Do  not  assume  that  all  respondents  will  have 

•  high-speed  internet  connections 

•  the  same  hardware  platforms 

•  the  same  computer  operating  systems 

•  the  same  types  and  versions  of  Internet  browsers 

The  appearance  and  operation  of  the  Web-based  questionnaire  needs  to 
be  consistent  across  a  variety  of  hardware  and  software  systems. 

If  respondents  must  meet  certain  system  requirements  to  use  the  Web- 
based  version  of  the  questionnaire,  you  must  make  this  absolutely 
clear  and  provide  alternatives  (such  as  a  paper-based  version  of  the 
questionnaire). 
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4.3.5  Web-Based  Questionnaires  (continued) 


Pre-testing  the 
implementation 


In  this  section  a  number  of  design  issues  were  discussed  that  should  be 
considered  in  a  Web-based  implementation  of  a  survey  questionnaire. 
While  many  of  these  guidelines  may  seem  intuitive,  your 
implementation  will  need  to  be  tested  thoroughly. 

When  testing,  remember  that  respondents  will  be  accessing  your 
questionnaire  using  different  platforms  and  operating  systems  and 
various  types  and  versions  of  Web  browsers.  Also  consider  the  impact 
of  network  firewalls.  Your  implementation  should  be  thoroughly  tested 
to  account  for  the  various  compatibility  issues  in  a  heterogeneous 
network  environment. 

Have  members  of  your  team  test  the  Web-based  version  of  the 
questionnaire.  Encourage  your  team  to  enter  unusual  inputs  to  find  out 
whether  the  system  can  handle  erroneous  entries.  Be  sure  that  answers 
are  not  lost  and  that  the  submit/save  function  works  properly. 


The  system  should  be  able  to  withstand  input  errors  without 
compromising  the  user’s  experience.  For  example,  the  system  should 
never  “lock-up”  due  to  mistaken  inputs,  nor  should  it  allow  for  invalid 
inputs  (e.g.,  allowing  two  radio  buttons  to  be  checked  when  your 
question  requires  a  single  input).  Also  ensure  that  navigational 
controls — either  the  browser’s  back-button  control  or  navigational 
controls  provided  within  the  questionnaire — do  not  cause  problems. 
Respondents  should  be  able  to  back-track  to  change  answers  (unless 
the  researcher  intentionally  prevents  this  of  j. 

as  part  of  the  question  strategy).  Q  ^ 

This  pre-test  of  the  implementation  is  not  f  r  C  £ 
the  final  test.  It  is  simply  meant  to  work  out  , 

obvious  bugs  in  the  implementation.  Stage  ' 

5,  Pilot  Test  Questionnaire,  is  a  critical  stage  I  f 

of  the  process  where  the  implementation,  in  «ft*vr  *  >- 

its  entirety,  is  tested  with  representative  Stress-test  your 

members  of  the  target  audience.  implementation 

to  get  out  the  bugs. 
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4.4 


Develop  Ancillary  Documents 


—  4.1 

—  4.2 

—  4.3 
— ■  4.4 


Determine  the  questions  to  be  asked 

Select  the  question  type/format  for  each  question  and 
specify  wording 

Design  the  question  sequence  and  overall  questionnaire  layout 
Develop  ancillary  documents 


Introduction  The  focus  of  stage  4  was  the  questionnaire  instrument  itself.  However, 

you  will  also  need  to  develop  other  ancillary  documents  that  are 
necessary  to  support  implementation  of  the  survey.  These  include: 

•  a  pre-notification  letter  (or  email) 

•  a  questionnaire  cover  letter  (for  paper-based  versions)  or  an 
introduction  screen  (for  Web-based  questionnaires) 

•  a  reminder  letters/emails  (to  remind  respondents  to  complete  the 
questionnaire  before  the  deadline) 

•  a  thank-you  letter  (or  email) 


Pre-notification 

letter 


However,  if  you  are  distributing  the  pre-notification  letter  by  postal 
mail,  you  need  to  consider  the  added  postage  cost.  A  study  by  Wright 
in  which  he  used  pre-notification  in  conjunction  with  reply  paid 
envelopes,  two  reminder  letters,  and  a  replacement  questionnaire 
found  that  pre-notification  did  not  increase  response  rates.  In  his  study, 
all  it  did  was  increase  cost  [Wright  95]. 

However,  if  you  are  using  email  rather  than  postal  mail,  the  additional 
cost  is  insignificant  and  there  is  no  reason  not  to  use  a  pre-notification 
letter  since  it  may  improve  your  response  rate. 

The  pre-notification  letter  covers  many  of  the  same  points  that  are 
covered  in  the  cover  letter  that  accompanies  the  questionnaire.  (See  the 
next  paragraph  on  cover  letters.)  In  addition,  it  tells  respondents  when 
they  will  receive  or  be  provided  access  to  the  survey  questionnaire. 


A  number  of  researchers  have  found  that  a  pre-notification  letter 
improves  survey  response  rates.  In  one  study,  an  8%  increase  in 
response  rate  was  reported  [Yu  83].  In  a  later  study,  Yammarino 
reported  an  average  increase  in  response  of  28.5%  after  the  use  of  a 
pre-notification  letter  [Yammarino  91]. 
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4.4  Develop  Ancillary  Documents  (continued) 


The  cover  letter 


Sponsor 


Incentives 


Introduction 

screen 

(for  Web-based 
questionnaires) 


The  cover  letter  should  include  concise  statements  of 

•  the  purpose  of  the  survey 

•  the  organization  or  individuals  sponsoring  the  survey 

•  encouragement  and/or  incentives  to  participate 

•  anonymity  or  confidentiality  (that  is,  assurance  that  the 
respondents  and  their  individual  responses  will  not  be  revealed, 
but  only  combined  with  many  other  responses  to  learn  about 
overall  opinions) 

•  a  list  of  the  sections  included  in  the  questionnaire 

•  an  explanation  of  how  the  results  will  be  published  and  how  the 
results  will  be  used 

•  and  estimate  of  how  long  it  will  take  to  complete  the  questionnaire 

•  the  calendar  deadline  for  completing  the  questionnaire 

•  a  phone  number  or  email  address  to  contact  if  the  recipient  has  any 
concerns  or  questions 

An  example  email  cover  letter  appears  in  Appendix  E  on  page  125. 


In  some  cases,  it  would  be  advantageous  to  have  the  sponsoring 
organization’s  leader  sign  the  cover  letter.  This  can  improve  response 
rates  by  lending  credibility  to  the  survey  project.  It  can  also 
communicate  a  sense  of  urgency  and  need  for  participation. 


Numerous  studies  have  demonstrated  that  incentives  will  improve 
survey  response  rate  [Linsky  75,  Nederhof  83,  Dommeyer  85,  Eisinger 
74,  Turley  99,  Gunn  81,  Robertson  78,  Brennan  92]. 

Some  examples  of  incentives  include  pens,  coffee  mugs,  and  t-shirts. 

If  you  believe  the  respondent  will  be  interested  in  the  results  of  the 
survey,  then  an  incentive  could  be  providing  a  copy  of  the  results. 


The  introduction  screen  of  a  Web-based  questionnaire  is  essentially  the 
cover  letter  for  the  Web-based  version.  The  content  (listed  above  in  the 
paragraph  called  “The  cover  letter”)  is  generally  the  same. 
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4.4  Develop  Ancillary  Documents  (continued) 


Reminder  letters 


Thank  you 
letter 


Conducting  team 
reviews 


Despite  people’s  best  intentions,  many  will  simply  forget  to  complete 
the  questionnaire  unless  they  are  reminded.  Following  distribution  of 
the  questionnaire,  a  reminder  letter  or  letters  can  be  sent  to  those 
respondents  who  have  not  completed  the  questionnaire. 

Research  has  clearly  demonstrated  that  reminder  letters  are  effective 
for  improving  response  rates.  Brennan  claims  that  the  most  basic 
approach  to  achieving  high  survey  response  rates  is  to  use  at  least  two 
reminders  (and  for  mail  surveys,  to  include  a  copy  of  the  questionnaire 
with  each  reminder  along  with  a  paid  reply  envelope)  [Brennan  92], 

Reminder  letters  should  be  brief  and  to  the  point.  For  Web-based 
versions,  provide  the  URL  for  accessing  the  questionnaire.  In  both 
cases,  it  is  a  good  idea  to  provide  contact  information  in  case  the 
respondent  has  questions  or  concerns  about  how  to  proceed. 

Keep  track  of  who  has  responded  and  who  has  not  and  only  send 
reminders  to  the  latter.  In  this  way,  you  avoid  annoying  those  who 
have  responded  with  inappropriate  email  messages. 


Consider  whether  you  want  to  send  respondents  a  brief  thank-you  note 
following  their  participation  in  your  survey.  If  you  are  providing 
incentives,  your  thank-you  note  can  be  sent  along  with  the  incentive. 


Make  sure  you  include  the  team  in  reviews  of  all  intermediate 
documents  created  during  the  development  process. 
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4.4  Develop  Ancillary  Documents  (continued) 
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5  Pilot  Test  Questionnaire 


Introduction  In  the  last  section,  a  number  of  design  issues  were  discussed  that 

should  be  considered  with  both  paper-based  and  Web-based 
implementations.  If  you  followed  these  guidelines,  you  might  feel 
confident  that  you  now  have  a  questionnaire  that  is  ready  to  be 
distributed  to  your  respondents.  Stop  right  there!  The  questionnaire  is 
not  ready  for  distribution.  While  you  may  have  followed  the 
prescribed  guidelines,  the  importance  of  empirical  verification  through 
thorough  testing  cannot  be  understated.  This  is  the  case  for  both  paper- 
based  questionnaires  and  Web-based  questionnaires. 

Up  to  this  point  in  the  survey  development  process,  you  have  likely 
invested  weeks  of  your  time  to  get  to  this  point  in  your  questionnaire 
development.  If  you  are  tempted  to  distribute  your  questionnaire 
without  testing,  then  try  to  imagine  your  horror  when,  after 
distributing  the  questionnaire,  you  are  bombarded  with  telephone  calls 
and  emails  from  respondents  who  were  not  able  to  successfully 
complete  your  questionnaire  due  to  interpretation  difficulties  or 
technical  problems.  That  is  not  a  pretty  picture,  is  it?  But  the  risk  is 
real  unless  you  take  the  time  to  test  your  questionnaire. 


What  is  a  pilot  A  pilot  test  is  a  simulation  of  your  survey  implementation  carried  out 
test?  on  a  small  scale  with  members  of  your  target  population. 


Team  reviews  are  not  considered  pilot  tests.  The  key  characteristic  of  a 
pilot  test  is  that  you  are  using  actual  members  of  your  target  audience 
to  test  your  implementation. 

This  certainly  does  not  imply  that  team  reviews  are  not  needed.  Team 
reviews  are  recommended  throughout  development  as  the  questions 
are  evolving.  In  fact,  it  is  a  good  idea  to  try  out  the  questions  on  others 
outside  the  team  to  identify  weaknesses  and  potential  difficulties. 

Continued  on  next  page 


Are  team  reviews 
considered  pilot 
tests? 
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5  Pilot  Test  Questionnaire  (continued) 


Are  team  reviews 
considered  pilot 
tests?  (continued) 


Purpose  of 
pilot  testing 


Testing  the 
questions 


Testing  for  layout 


Of  course,  before  conducting  a  pilot  test,  a  good  method  for 
identifying  difficulties  is  for  the  research  team  to  serve  as  respondents 
in  a  pre-test  of  the  questionnaire.  This  was  discussed  on  page  70. 

“The  questionnaire  designer  must  understand  the  need  to  pretest, 
pretest,  and  then  pretest  some  more.” 

— Section  on  Survey  Research  Methods 
American  Statistical  Association 
[ASA  99] 


Pilot  tests  are  conducted  to  expose  problems  or  weaknesses  in  the 

•  questions 

•  questionnaire  layout 

•  process 

•  technology  (if  a  Web-based  questionnaire  is  used) 


During  a  pilot  test,  you  should  evaluate  the  questions  to  determine  if 

•  there  is  confusion  resulting  from  the  instructions 

•  the  right  questions  are  being  asked 

•  the  questions  are  understandable 

•  the  order  of  the  questions  seems  logical 

•  answer  choices  (for  closed-ended  questions)  are  appropriate  and 
no  answer  choices  have  been  overlooked 

•  there  are  any  terms  that  are  not  understood  or  are  open  to  multiple 
interpretations 


During  a  pilot  test,  you  should  evaluate  the  layout  to  determine  if 

•  there  is  any  difficulty  navigating  through  the  questionnaire 

•  placement  of  questions  (on  paper  or  on  the  computer  screen)  is 
optimal 

•  font  sizes  are  easy  to  read 

•  conventions  with  the  use  of  bold-face,  italics,  color,  and  so  forth 
were  consistent  and  helped  the  respondent  (That  is,  did  these 
conventions  work  as  intended  by  the  questionnaire  designers?) 


76 


CMU/SEI-2005-HB-004 


5  Pilot  Test  Questionnaire  (continued) 


Testing  the 
process 


Testing  for 
problems  with  the 
technology 


Approach 


During  a  pilot  test,  you  should  evaluate  the  process  to  determine 

•  whether  the  survey  objectives  are  meaningful  and  of  interest  to  the 
pilot  test  respondents 

•  how  interested  the  respondents  are  (personally)  in  the  results  for 
the  survey 

•  how  long  it  takes  respondents  to  complete  the  questionnaire 

•  how  the  respondents  felt  about  the  length  of  time  to  complete  the 
questionnaire 

•  overall  satisfaction  with  the  survey  process 

Also,  ask  your  pilot  participants  their  opinions  of  what  the  duration  of 

the  response  window  should  be  (that  is,  the  time  between  the  release  of 

the  questionnaire  to  all  respondents  and  the  deadline  for  completion). 


If  you  are  using  a  Web-based  questionnaire,  you  should  evaluate  the 
technology  to  determine  if 

•  erroneous  input  caused  a  problem 

•  respondents  were  able  to  navigate  easily  from  screen  to  screen 

•  error  messages  were  perceived  as  friendly  and  helpful 

•  there  were  lag  times  that  caused  the  respondent  to  wait  before 
performing  a  task  in  the  questionnaire 

•  input  data  was  correctly  recorded  in  the  database 

•  data  was  lost  because  it  wasn’t  saved  properly 


The  approach  you  take  to  the  pilot  test  will  depend  on  whether  or  not 
the  participants  are  on-location.  Here  are  some  example  test  scenarios 
that  can  be  used  to  conduct  the  pilot  test. 

When  participants  are  on  location 

It  is  best  if  all  pilot  participants  are  in  the  same  room.  The  research 
team  should  observe  the  respondents  completing  the  questionnaire. 

•  Begin  the  pilot  test  by  explaining  the  process  to  the  participants. 

•  Take  detailed  notes,  keeping  an  eye  out  for  signs  that  individuals 
are  experiencing  any  difficulties. 

•  Request  written  comments  about  the  survey  (e.g.,  ask  participants 
how  appropriate  the  wording  of  the  questions  was).  Consider 
preparing  an  evaluation  form  that  asks  your  testers  to  answer 
questions  about  specific  aspects  of  their  experience. 

•  After  participants  have  completed  the  questionnaire,  hold  a 
discussion  and  record  all  feedback. 


Continued  on  next  page 
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5  Pilot  Test  Questionnaire  (continued) 


Approach 

(continued) 


Using  the 
feedback 


When  participants  are  off-site 

In  some  cases,  pilot  participants  will  be  geographically  dispersed  and 
you  will  not  be  able  to  observe  them  directly.  Therefore,  you  will  need 
to  prepare  clear  written  instructions  and  an  evaluation  form  that  can 
serve  as  stand-alone  support  for  conducting  the  pilot  test. 

•  The  instructions  should  specify  the  type  of  feedback  you  are 
looking  for.  (You  can  use  the  information  provided  within  this 
section  to  help  you  determine  the  type  of  feedback  you  want.) 

•  The  evaluation  form  should  capture  any  and  all  issues  the 
participants  experienced  while  they  completed  the  questionnaire. 

•  Instruct  the  participant  to  track  questionnaire  completion  time 
separately  from  the  time  required  to  make  notes  on  evaluation 
form. 

•  After  completion  of  the  pilot  test,  collect  the  evaluation  forms  and 
schedule  follow-up  phone  calls  to  ask  clarifying  questions  about 
the  participant’s  experience. 


Based  on  the  outcome  of  the  pilot  test,  make  the  necessary 
improvements  to  your  questionnaire  and  implementation  alternatives. 
If  you  have  done  a  good  job  of  pre-testing  the  implementation  (with 
members  of  your  team  and  other  colleagues),  then  any  changes 
required  should  be  fairly  minor. 
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6  Distribute  the  Questionnaire 


Introduction  Having  thoroughly  tested  your  questionnaire,  you  are  now  ready  to 

distribute  it  or  provide  access  to  it. 

The  respondents  should  have  been  pre-notified  of  your  desire  for  their 
participation  by  the  time  you  are  ready  to  distribute  the  questionnaire. 
(See  page  71  for  a  discussion  of  the  pre-notification  letter  and  its 
benefits.)  Therefore,  it  should  be  no  surprise  to  the  respondents  when 
they  finally  receive  the  questionnaire  or  access  to  a  Web-based  version 
of  the  questionnaire. 

This  section  discusses: 

•  controlling  distribution 

•  deciding  the  duration  of  the  response  window 

•  monitoring  the  response  rate 

•  sending  reminders  to  respondents 

To  protect  the  validity  of  your  research  approach,  you  must  take 
precautions  so  that  a  questionnaire  form  cannot  be  duplicated  and 
submitted  multiple  times  by  the  same  individual,  or  submitted  by  an 
unauthorized  individual. 

For  paper-based  questionnaires  that  are  mailed  or  emailed: 

•  Print  a  unique  alphanumeric  identifier  on  each  questionnaire 
distributed.  Keep  a  list  that  tracks  (a)  respondent  name,  and  (b) 
questionnaire  identifier  number.  The  alphanumeric  identifier  is  for 
administrative  purposes  only  and  can  be  used  for  different  reasons 
(e.g.,  to  avoid  sending  needless  reminders  to  individuals  who  have 
already  responded  to  the  survey). 

•  If  the  questionnaire  is  emailed  as  an  attachment,  send  a  version  of 
the  document  that  cannot  be  edited  (e.g.,  an  Adobe  Acrobat  with 
security  set  so  that  the  file  is  not  editable). 

For  Web-based  versions  of  questionnaires,  control  access  to  the 
questionnaire  by  providing  a  unique  PIN  or  usemame/password 
combination.  (This  was  discussed  briefly  on  page  64.) 


Controlling 

distribution 
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6  Distribute  the  Questionnaire  (continued) 


Duration  of 
the  response 
window 


Monitoring  the 
response  rate 


The  response  window  is  the  duration  between  the  time  the 
questionnaire  is  distributed  and  the  deadline  for  completing  the 
questionnaire. 

As  you  already  know,  a  low  level  of  response  to  your  survey  will 
weaken  the  study.  To  maximize  the  response  rate,  ensure  that  you  are 
providing  a  reasonable  amount  of  time  for  individuals  to  respond. 

How  long  should  the  response  window  be?  In  most  cases,  three  to  five 
weeks  should  be  sufficient.  But  use  your  best  judgment  and  take  into 
account  characteristics  of  your  target  population. 

Here  are  some  things  to  think  about: 

•  Can  your  audience  analysis  inform  you  about  characteristics  of 
your  respondents  that  will  affect  their  ability  to  work  on  the 
questionnaire  in  a  timely  manner?  Ask  pilot  participants  what  their 
opinion  was  of  the  duration  of  the  response  window  (see  the 
paragraph  labeled  ‘Testing  the 

process”  on  page  96). 

•  If  you  are  releasing  the  questionnaire  during  the  summer  months, 
be  aware  that  many  individuals  reserve  significant  time  for 
vacation  (sometimes  more  than  three  weeks).  During  this  time, 
individuals  are  away  from  their  offices  and  their  computers. 
Likewise,  consider  the  impact  of  holidays,  especially  those  toward 
the  end  and  beginning  of  the  calendar  year. 


Keep  track  of  the  response  rate  after  you  provide  access  to  the 
questionnaire.  This  will  help  you  know  when  you  should  send 
reminders.  Keeping  track  of  information  such  as  this  can  also  help  you 
plan  future  surveys. 

Below  is  an  example  response-rate  graph: 


Cumulative 
number  of 
questionnaires 
completed 
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6  Distribute  the  Questionnaire  (continued) 


When  to  send 
reminder  letters 


Timing  the  dispatch  of  reminder  letters  depends  on  the  response 
timeframe  you  have  established  and  the  response  rate  you  observe 
after  you  have  provided  access  to  the  questionnaire. 

For  a  two-month  response  timeframe  (that  is,  respondents  have  two 
months  to  complete  the  questionnaire  before  the  deadline),  you  may 
want  to  send  a  reminder  after  three,  five,  and  seven  weeks.  If  you  are 
using  a  paper  questionnaire  sent  through  the  postal  mail,  then  you  may 
only  want  to  send  out  two  reminders  (due  to  cost)  during  the  fourth 
and  sixth  week  after  sending  the  original  questionnaire.  With  emailed 
reminders,  cost  is  not  a  factor. 
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6  Distribute  the  Questionnaire  (continued) 
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7  Analyze  Results  and  Write  Report 
Overview 


1.  Identify  the  research 
objectives 


3.  Design  sampling 
plan 


5.  Pilot  test 
questionnaire 


2.  Identify  &  characterize 
target  audience 


4.  Design  &  write 
questionnaire 


6.  Distribute 
questionnaire 


7.  Analyze  results 
and  write  report 


7.1 

7.2 


Analyze  the  results 
Write  the  report 


Introduction  At  this  point,  the  questionnaire  completion  deadline  has  been  met  and 

you  have  all  of  the  response  data.  It  has  been  a  lot  of  work  to  get  here, 
but  the  work  is  far  from  done. 

Now  comes  the  task  of  pulling  the  information  together  to  make 
observations,  conduct  analyses,  make  interpretations,  and  draw 
conclusions.  The  vehicle  for  communicating  all  of  this  information  is 
the  research  report. 

The  first  task  ahead  of  you  is  to  compile  the  questionnaire  data  and 
transform  it  into  meaningful  representations  that  can  be  interpreted  and 
analyzed.  As  a  result  of  the  analysis,  observations  are  made,  leading 
the  researchers  to  form  interpretations  and  draw  conclusions.  This  is 
discussed  in  7.1,  “Analyze  the  results.” 

The  second  task  is  to  package  the  analysis  as  an  information  product 
that  can  be  interpreted  and  used  by  others.  This  is  the  focus  of 
subsection  7.2,  “Write  the  report.” 

Your  report  could  take  a  number  of  different  forms,  depending  on  the 
particular  audience  you  are  addressing.  A  narrative  document 
(including  graphical  depictions  of  the  results)  will  be  your  key 
information  product.  But,  it  is  also  likely  that  you  will  develop 
presentations  that  address  various  aspects  of  your  research  work.  And 
if  your  work  is  of  interest  to  the  general  community,  you  may  want  to 
write  journal  articles  to  report  your  research  findings. 


In  this 
section 


Section 

See  page 

Analyze  the  results 

84 

Write  the  report 

98 
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7.1 


Analyze  the  Results 


Introduction  In  this  subsection,  concepts  related  to  measurement  are  discussed  and 

guidance  is  provided  for  analyzing  the  survey  results. 


If  you  are  using 
a  Web-based 
questionnaire 


One  of  the  major  benefits  of  using  Web-based  questionnaires  is  the 
ease  of  data  collection  and  data  organization:  it  is  all  automated.  If  you 
are  using  COTS  survey  software,  then  the  capability  to  perform  these 
tasks  is  embedded  in  the  software.  However,  it  is  still  necessary  to 
have  a  thorough  understanding  of  how  the  software  collects  and 
manipulates  the  data. 

If  you  are  using  a  Web-based  questionnaire,  you  should  have  already 
tested  the  collection  and  organization  of  the  response  information. 
Also,  the  export  capabilities  of  the  survey  application  should  have 
been  thoroughly  tested  to  ensure  accurate  transfer  of  respondent  data 
to  spreadsheet  programs  or  specialized  statistical  packages  for  further 
analysis  and  graphical  display. 


If  you  are  using  a  If  you  are  using  a  paper-based  version  of  the  questionnaire  that  has 
paper-based  been  mailed  or  emailed  to  respondents,  someone  will  need  to  enter  the 

questionnaire  data  into  the  computer  manually. 

As  a  precursor  to  analysis,  the  data  must  be 

•  coded,  and  then 

•  organized  into  a  spreadsheet  or  database 

In  the  case  of  closed-ended  questions  in  your  questionnaire,  decisions 
about  coding  were  made  when  you  were  constructing  the  combination 
of  question  with  response  format  during  question  construction  (see 
subsection  4.2.1  beginning  on  page  42). 

There  are  a  number  of  measurement  issues  related  to  the  coding  of 
response  information  and  its  further  quantitative  analysis,  which  are 
discussed  in  the  next  subsection. 
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7.1  Analyze  the  Results  (continued) 


Coding 


Nominal 

numbers 


Coding  is  the  assignment  of  a  number  value  to  text.  Text  responses  are 
coded  to  ease  data  entry,  facilitate  data  manipulation,  and  allow 
quantitative  analysis.  There  are  various  ways  to  assign  numbers  to 
questionnaire  answer  choices.  The  way  numbers  are  assigned  matters 
because  the  type  of  assignment  determines  the  type  of  measurement 
analysis  that  can  be  undertaken. 

The  assignment  types  are  classified  in  terms  of 
the  level  of  measurement  used  in  a  question’s 
answer  format.  They  are 

•  nominal 

•  ordinal 

•  interval 

•  ratio 


In  a  nominal  response  format,  the  numbers  used  have  no  meaning 
except  as  a  placeholder  for  the  text  response.  The  numbers  represent 
categories.  For  example,  arbitrary  number  assignments  could  be  made 
for  closed-ended  questions  with  unordered  responses  (see  page  42  for 
more  information  about  questions  with  answer  choices  of  this  type). 

An  example  of  nominal  response  type  assignment  of  numbers  is 
shown  below. 


Nominal 

Response  number  assignment 

Program  Manager  1 

Project  Manager  2 

Engineer  3 

Clerical  4 

Other  5 

Nominal  number  assignments  are  likely  a  holdover  from  the  days 
when  responses  were  transferred  to  punch  cards  for  computer  analysis. 
Using  today’s  desktop  computer  applications,  nominal  number 
assignments  are  rarely  used. 
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7.1  Analyze  the  Results  (continued) 


Ordinal  type 

number 

assignments 


An  ordinal  response  format  asks  respondents  to  rank-order  their 
preferences  in  a  list.  The  number  indicates  a  place  or  position  within 
the  list  of  items. 

An  example  of  ordinal  response  numbers  is  shown  below. 

Q7.  Rank  the  importance  of  the  following  issues  to  your 
job  satisfaction  by  placing  a  number  next  to  each 
item  (1  =  most  important,  4  =  least  important). 

Education  opportunities 
Flex-time  schedule 
Compensation 
Friendliness  of  workplace 


□ 

□ 

□ 

□ 


Measurements  with  ordinal  scales  are  ordered  in  the  sense  that  each 
number  has  a  value  in  relationship  to  the  other.  However,  the  intervals 
between  the  numbers  are  not  necessarily  equal.  For  example,  on  the 
four-point  rating  scale  measuring  attitudes  toward  job  satisfaction,  the 
difference  between  a  rating  of  1  and  a  rating  of  2  might  not  represent 
the  same  difference  as  the  difference  between  a  rating  of  3  and  a  rating 
of  4.  There  is  no  true  zero  point  for  ordinal  scales  since  the  zero  point 
is  chosen  arbitrarily.  The  lowest  point  on  the  rating  scale  in  the 
example  above  was  arbitrarily  chosen  to  be  1.  It  could  just  as  well 
have  been  0  or  -4. 

Therefore,  it  would  be  wrong  to  calculate  the  average  of  the  ratings  for 
one  of  the  listed  items.  The  result  would  make  no  sense.  Ordinal 
numbers  cannot  be  analyzed  by  calculating  averages.  The  correct  way 
to  report  the  results  would  be  to  report  frequencies  of  how  each  item 
was  ordered  in  a  particular  way.  Median  values  can  be  reported,  but 
not  averages. 
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7.1  Analyze  the  Results  (continued) 


Likert  scales 
are  ordinal 


Interval  number 
assignments 


Ratio  type 


Another  ordinal  response  scale  was  discussed  on  page  45 — the 
Likert  scale. 

Because  Likert  scales  are  ordinal,  it  would  be  wrong  to  make  the 
assignments  listed  below  with  the  intention  of  combining  and 
manipulating  the  assigned  numbers  to  form  averages.  Likert  scale 
response  information  must  be  reported  as  frequencies  within  each 
category. 


Response  Scale 


Strongly  disagree  1 

Disagree  2 

Undecided  3 

Agree  4 

Strongly  Agree  -  5 


With  interval  numbers,  the  distance  between  attributes  does  have 
meaning  (although  the  scale  does  not  possess  an  absolute  zero  point). 

It  is  not  generally  possible  to  develop  closed-ended  questionnaire 
responses  that  map  to  an  interval  scale.  Response  scales  are  typically 
ordinal  in  nature. 


Ratio-type  response  information  can  be  averaged  and  formed  into 
ratios.  With  ratio  numbers,  there  is  an  absolute  zero.  This  means  that 
you  can  construct  meaningful  fractions  with  ratio  numbers.  In  survey 
research,  ratio  numbers  are  found  with  count  numbers.  For  example, 
when  you  ask  the  respondent  “how  many?”  types  of  questions. 

An  example  of  ratio-type  response  numbers  is  shown  below. 

Q3.  List  the  total  number  of  on-board  full-time 

equivalents  (FTE)  for  FY2005  in  your  organization, 
including  both  Government  and  Contractor 
personnel.  ' 
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7.1  Analyze  the  Results  (continued) 


Organizing  If  you  are  using  hard  copy  forms  of  the  questionnaire,  it  will  be 

your  information  necessary  to  organize  the  response  information  into  a  spreadsheet  or 
database.  Many  spreadsheet  programs  provide  powerful  native  tools 
for  statistical  analysis  and  graphical  depiction.  Also,  most 
spreadsheets  can  export  the  spreadsheet  information  to  computer 
applications  that  focus  on  statistical  analysis. 

The  easiest  way  to  do  this  is  to  organize  a  spreadsheet  area  with  each 
row  corresponding  to  a  single  respondent  and  the  columns 
representing  the  specific  answer  choice  variables.  For  a  questionnaire 
that  has  ten  questions  with  each  question  having  five  possible  answer 
choices,  the  number  of  columns  would  be  expressed  by 

Number  of  columns  =  10  *  5  =  50  columns 


If  there  are  500  respondents  in  the  sample,  then  the  table  would  be  a 
500  row  by  50  column  matrix. 

The  example  below  depicts  part  of  a  spreadsheet  matrix.  The  letters 
represent  answer  choices  for  each  question.  The  white  part  of  the 
matrix  is  where  the  questionnaire  results  would  be  entered.  The 
number  “1”  is  entered  for  a  response  alternative  selected  by  a  given 
respondent. 


Question  #1 

Question  #2 

••• 

Question  #n 

a. 

b. 

3 

e. 

a. 

3 

c. 

••• 

a. 

b. 

c. 

d. 

e. 

Respondent  #1 

1 

_ 

■ 

1 

■ 

M 

1 

m 

1 

_ 

Respondent  #2 

■ 

i 

■ 

■ 

■ 

■ 

0 

1 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

0 

■ 

■ 

■ 

■ 

Respondent  #n 

■ 

■ 

i 

1 

m 

■ 

.mu 

■ 

« 

■ 

■ 

Statistical 
analysis  of 
the  results 


Once  the  data  has  been  organized  appropriately,  statistical  analysis  can 
be  conducted.  Statistical  analysis  for  survey  data  examines  response 
patterns — frequencies  of  different  responses,  what  response  occurred 
most  frequently  for  each  question  within  a  group,  variation  in 
responses  within  a  group,  and  differences  in  ways  different  subgroups 
(within  the  same  survey)  responded.11 


11  Subgroup  analysis  is  possible  using  cross-tabulation  discussed  on  page  9. 
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7.1  Analyze  the  Results  (continued) 


Displaying 
the  data 


Graphical 

presentation 


Examples  of 
charts  and  graphs 
used  in  surveys 


The  first  task  of  the  analysis  is  to  determine  the  frequency  distribution 
of  responses  for  each  question.  The  term  “distribution”  means  the 
frequency  of  each  value  selected  by  the  respondents  over  the  range  of 
all  valid  values.  There  are  essentially  two  ways  to  present  data  to 
support  analysis: 

•  graphical  display  (charts  and  graphs) 

•  tables 

For  large  amounts  of  response  data  (which  typically  result  from 
surveys),  you  will  want  to  display  your  frequency  distributions 
graphically.  Graphical  presentations  are  covered  in  the  following 
paragraphs.  There  are  situations  when  tabular  information  is  preferred, 
which  are  discussed  briefly  after  issues  about  graphical  presentations 
are  covered. 


Quantitative  research  is  about  collecting,  analyzing,  and  interpreting 
numbers.  However,  the  human  mind  is  rather  limited  when  it  comes  to 
understanding  patterns  within  lists  of  raw  numbers.  In  order  to  make 
sense  of  large  volumes  of  data,  the  raw  numbers  need  to  be 
transformed  into  something  intelligible,  such  as  a  graph  or  picture. 

With  today’s  technology,  almost  any  type  of  data  graph  can  be 
displayed  with  standard  applications  available  on  every  desktop. 
However,  it  is  still  up  to  the  human-in-the-loop  to  ensure  good  design. 

What  is  good  design?  Good  design  supports  the  data  and  the  analysis 
of  the  data.  Effective  presentation  of  data  is  an  art  form.  A  well- 
designed  graph  presents  data  clearly  and  accurately.  It  exposes  the 
data’s  features  of  interest — that  is,  the  message  within  the  data. 

A  complete  treatment  of  the  development  of  graphs  is  beyond  the 
scope  of  this  document.  However,  a  number  of  guidelines  are 
presented  in  the  following  paragraphs  as  tips  for  developing  effective 
graphs  of  your  data. 


The  right  chart  can  facilitate  analysis  and  interpretation.  While  there 
are  many  different  ways  to  display  data,  some  of  the  more  popular 
ways  include  bar  charts  (both  vertical  and  horizontal),  pie  charts,  and, 
to  a  lesser  extent,  line  graphs  and  box  plots.  While  these  types  of 
charts  and  graphs  do  not  represent  all  the  ways  that  data  can  be 
presented,  they  are  the  predominant  methods  used  but  for  survey  data. 
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7.1  Analyze  the  Results  (continued) 


Vertical  bar 
chart 


The  vertical  bar  chart  is  sometimes  called  a  column  chart.  It  is  the 
most  commonly  used  method  for  comparing  the  frequency  or 
magnitude  of  the  that  appear  along  the  horizontal  axis.  The  length  of 
the  bar  is  proportional  to  the  value  it  depicts.  Bar  charts  can  also  be 
used  to  depict  time  series  data  when  the  number  of  time  intervals  is 
small. 

This  is  an  example  of  a  basic  vertical  bar  chart. 


Response 


Stacked 
vertical  bar 
chart 


The  basic  vertical  bar  chart  can  be  varied  in  a  number  of  ways.  One 
variation  is  the  stacked  vertical  bar  chart.  The  stacked  bar  chart 
consists  of  one  or  more  segmented  bars,  with  each  segment  of  a  bar 
representing  the  relative  share  of  a  total  that  the  component 
contributes. 

Subdivided  bar  charts  are  not  a  preferred  format  [Rutchik  98].  They 
should  be  used  with  caution.  It  is  difficult  to  make  comparisons  among 
the  second,  third,  or  subsequent  segments  in  a  stacked  bar  since  the 
comparison  is  not  being  made  from  a  common  base. 

Below  is  an  example  of  a  stacked  vertical  bar  chart. 


D  Strongly  Disagree 

■  No  Response 

□  Disagree 

□  Don!  Know  or  N/A 

■  Strongly  Agree 

D  Agree  | 
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7.1  Analyze  the  Results  (continued) 


Horizontal  bar  Horizontal  bar  charts  are  like  vertical  bar  charts  turned  sideways.  One 

charts  advantage  of  this  presentation  is  the  extra  room  on  the  vertical  axis  for 

labels.  This  is  especially  important  when  the  number  of  bars  to  be 
portrayed  is  relatively  large.  Also,  there  is  a  better  sense  of  continuity 
when  a  horizontal  label  is  combined  with  a  horizontal  bar  [Chambers 
83,  Schmid  83]. 

Below  is  an  example  of  an  ordered  horizontal  bar  chart 


Technical  Solution 
Software  Architecture  Development  and  Assessment 

Integrated  Teaming 
Project  Monitoring  and  Control 
Product  Integration 
System  Developer  shares  all  relevant  information 
Configuration  Management 
Project  Planning 
Supplier  or  Subcontract  Management 
Defined  processes 
Verification 
Requirements  Development 
Requirements  Management 
Validation 

Measurement  Analysis  and  Reporting 
Causal  analysis  and  resolution  for  defect  prevention 

Risk  Management 


0.0  1.0  2.0  3.0  4.0 


Weighted  Response 
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7.1  Analyze  the  Results  (continued) 


Pie  chart 


A  pie  chart  is  a  circular  diagram  divided  into  segments,  with  each  segment 
representing  a  category  or  subset  of  data.  The  value  or  amount  of  each 
category  is  proportional  to  the  size  of  the  sector.  The  total  area  of  the  circle 
represents  the  entire  sample  being  shown. 

Pie  charts  are  popular  because  they  are  simple  and  are  sufficient  to  display 
simple  messages.  But,  they  have  limited  utility  [Rutchik  98].  Tufte  states  that 
“the  only  worse  design  than  a  pie  chart  is  several  of  them”  [Tufte  83]. 

Despite  their  popularity,  pie  charts  should  be  used  sparingly  for  two  reasons. 
First,  they  should  only  be  considered  for  displaying  statistical  information 
when  there  are  no  more  than  six  components;  otherwise,  the  resulting  picture 
will  be  too  complex  to  understand.  Second,  pie  charts  are  not  useful  when  the 
components  have  similar  values  because  it  is  difficult  to  see  the  differences 
between  slice  sizes. 

An  example  of  a  pie  chart  is  shown  below.  (There  are  many  other  variations  of 
the  “pie  chart”  theme.) 
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7.1  Analyze  the  Results  (continued) 


Other  types  of 
graphs  &  charts 


Misuse  of 
graphics 


Guidelines  for 
effective  graphs 
&  charts 


Some  of  the  common  ways  to  display  survey  data  graphically  have 
been  described  briefly.  However,  there  are  other  valid  ways  that  might 
be  more  suitable  depending  on  the  features  you  are  trying  to  expose 
within  your  data  set. 

For  example,  a  line  graph  can  show  a  visual  comparison  between  two 
survey  variables  better  than  a  bar  chart.  In  some  cases,  a  box  plot  can 
be  very  descriptive  for  showing  the  distribution  of  a  survey  variable 
that  is  continuous  in  nature  [Tukey  77].  To  read  more  about  the  use  of 
box  plots,  see  Hayes  [Hayes  95]. 

You  should  explore  other  methods  that  may  be  useful  for  describing 
your  survey  data.  Experiment  with  different  ways  of  plotting  the  data 
to  see  what  works  best  for  you  [Tufte  83]. 


Graphs  and  charts  are  often  misused.  The  honest  researcher  must  have 
a  good  handle  on  how  graphs  can  be  used  to  mislead  rather  than 
inform.  Huff  shows  a  good  selection  of  methods  that  can  be  used  to 
make  numbers  say  something  different  from  what  the  actual  data 
supports.  He  looks  at  graphical  presentation,  correlations,  collection  of 
data,  sample  size,  sample  bias,  and  other  critical  factors  that  can  enter 
into  the  validity  of  statistics  and  of  the  conclusions  we  try  to  draw 
from  them  [Huff  54].  Tufte  also  describes  the  many  ways  charts  and 
graphs  can  mislead  through  poor  design  [Tufte  83], 


Detailed  design  guidance  is  beyond  the  scope  of  this  document.  To 
find  out  more  about  this  topic,  review  the  words  of  Kosslyn, 
Cleveland,  and  Tufte  for  excellent  treatment  of  the  issues  involved 
with  effective  graph  and  chart  design  [Kosslyn  93,  Cleveland  94,  Tufte 
83]. 

The  following  is  a  summary  of  some  of  key  guidelines  from  these 
authors: 

•  From  a  design  perspective,  keep  the  graph  as  simple  as  possible 
for  the  analysis  purpose  at  hand.  Do  not  use  three-dimensional  (3- 
D)  effects  or  fancy  crosshatching.  They  often  distort  or  hide  data, 
making  comprehension  difficult  or  impossible. 

•  Always  include  the  total  sample  size  as  part  of  the  graph  or  chart. 

•  Don’t  settle  for  the  default  graph  that  is  provided  by  a  software 
application.  Try  different  types  of  graphs  and  charts  and  choose 
the  one  that  works  best. 

•  Comparing  the  size  of  bars  in  a  bar  chart  is  easier  than  comparing 
slices  in  a  pie  chart. 


Continued  on  next  page 
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7.1  Analyze  the  Results  (continued) 


Guidelines  for 
effective  graphs 
&  charts 
(continued) 


When  to  use 
tables 


•  Use  scaling  consistently  within  a  graph  or  chart  and  across  graphs 
and  charts  that  will  be  compared.  Use  equal  increments  if  possible. 

•  Clearly  label  the  X  and  Y  axes  and  including  the  unit  that  is  being 
measured. 

•  Use  conventional  graphing  methods.  For  example,  time  should  be 
plotted  along  the  X  axis. 

•  Use  compatible  colors  within  charts  and  don’t  use  too  many 
colors.  Be  consistent  with  the  use  of  colors  and  fonts  in  a  series  of 
charts  of  graphs. 

•  Test  the  colors  you  select  for  graphs  and  charts  with  your  printer. 
The  colors  do  not  always  print  as  viewed  on  your  screen.  Consider 
whether  you  or  others  will  print  your  color  graphics  on  a  black  and 
white  printer.  If  this  is  the  case,  you  need  to  test  your  color  hues  to 
ensure  that  they  can  be  discriminated  after  printing  grayscale  or 
black  and  white. 

•  Use  rectangular  plots  rather  than  square.  The  human  eye  seems  to 
be  trained  to  look  for  horizontal  effects.  It  tends  to  scan  side-to- 
side  better  than  up-and-down  [Tufte  83]. 

In  almost  all  cases,  use  a  zero  baseline  as  the  anchor  of  your  Y  axis 
scale  since  the  vertical  scale  represents  size  or  amount.  The  design 
principle  at  operation  is  that  the  area  displayed  is  proportional  to  the 
quantity  of  interest.  If  the  scale  is  not  baselined  at  zero,  then  the 
proportions  will  be  hidden  from  view. 


In  some  instances  a  data  table  is  better  than  a  graphical  display.  A 
small  table  sometimes  can  be  more  effective  than  a  large  graph. 

In  general,  tables  are  better  than  graphs  for  giving  precise,  structured 
numeric  information,  whereas  graphs  are  better  for  indicating  trends 
and  making  broad  comparisons  or  showing  relationships. 

Tables  are  also  better  for  making  localized  comparisons  between  parts 
of  a  data  set.  In  some  cases,  you  might  want  to  include  a  table  to 
complement  a  graph  of  the  same  information. 

Ehrenberg  states  that  tables  should  be  used  instead  of  graphics  when 
you  have  20  or  fewer  numbers,  while  Zawitz  suggests  that  tables 
should  be  used  when  you  have  10  or  fewer  numbers. 

Ehrenberg  and  Zawitz  provide  other  helpful  suggestions  for  making 
tabular  presentations  effective  [Ehrenberg  81,  Zawitz  00]. 
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7.1  Analyze  the  Results  (continued) 


“Cross-tabulation”  is  a  method  used  to  make  comparisons  between 
two  or  more  nominal  variables.  Cross-tabulation  can  be  used  when  you 
want  to  know  how  categories  of  respondents  answered  the  same 
question. 

In  some  cases,  cross-tabulation  is  a  strategy  that  the  research  team 
planned  a  priori  for  evaluating  the  survey  results.  In  other  instances, 
results  are  cross-tabulated  due  to  interesting  questions  that  surfaced  as 
a  consequence  of  initial  analysis  of  the  survey  results. 

For  example,  consider  a  survey  that  includes  executives,  project 
managers,  engineering  staff,  and  clerical  staff.  You  might  want  to 
compare  their  answers  to  specific  questions  to  determine  if  there  are 
differences  based  on  an  individual’s  position  in  the  organization. 


When  you  cross-tabulate  to  examine  differences  between  two 
subgroups  in  your  survey,  you  might  want  to  test  to  find  out  if  the 
differences  are  statistically  significant. 

Various  statistical  tools  can  be  used  to  determine  whether  the 
differences  are  real  and  not  simply  due  to  sampling  error  associated 
with  the  precision  specified  in  the  sampling  plan.  If  the  test  determines 
that  a  difference  is  larger  than  what  might  have  occurred  by  chance, 
the  difference  is  said  to  be  “statistically  significant.” 

Developing  the  appropriate  statistical  tests  requires  knowledge  of 
statistical  concepts  and  theory  that  is  beyond  the  scope  of  this 
document.  For  an  excellent  treatment  of  this  topic,  consider 
publications  by  Backstrom  and  Williams  [Backstrom  81,  Williams  79]. 


Statistical 

significance 

between 

subgroups 


Cross-tabulation: 
deeper  analysis 
of  your  data 


Is  this  difference  significant 
or  due  to  sampling  error? 
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7.1  Analyze  the  Results  (continued) 


What  about  the 

open-ended 

questions? 


As  stated  previously  on  page  40,  open-ended  questions  should  be  used 
sparingly  and  carefully  in  self-administered  surveys.  Long  narrative 
responses  to  questions  are  very  difficult  and  sometimes  impossible  to 
interpret  and  quantify.  However,  under  certain  circumstances,  you  may 
want  to  include  one  or  two  as  part  of  your  questionnaire. 

Responses  to  open-ended  questions  must  be  processed  manually, 
which  is  very  time-consuming.  The  researcher  must  examine  each 
response,  interpret  it,  and  then  apply  judgment  to  filter  the  response 
into  categories  that  emerge  as  additional  responses  are  analyzed. 
Filtering  and  categorization  can  corrupt  the  original  intention  of  the 
respondent  and  that  is,  of  course,  a  threat  to  validity. 


Approach: 
categorizing 
responses  to 
open-ended 
questions 


The  Affinity  Diagram  Technique  is  a  tool  that  gathers  large  amounts  of 
language  data  (e.g.,  ideas,  opinions,  issues),  organizes  it  into  groupings 
based  on  the  natural  relationship  between  each  item,  and  defines 
groups  of  items  [Brassard  96].  One  of  the  best  ways  to  analyze 
responses  to  open-ended  questions  is  through  a  structured  team 
approach  using  an  adaptation  of  the  technique.  In  this  way, 
categorization  of  open-ended  response  information  does  not  rely  on  a 
single  person’s  opinion  but  is  arrived  at  through  team  consensus.  Steps 
for  this  technique  are  listed  below. 

Set-up 

1 .  Each  narrative  response  is  placed  in  a  bordered  table  cell.  The 
font  size  should  be  increased  to  14-16  points  so  the  responses  can 
be  read  by  a  group  of  people  standing  at  a  distance. 

2.  Use  a  paper  cutter  to  separate  each  response  from  the  other. 

3.  Position  the  responses  on  a  large  conference  table  (or  on  more 
than  one  table). 

Arrange  the  Responses  into  Related  Groupings 

4.  Look  for  two  responses  that  seem  to  be  related  in  some  way. 

Place  those  two  side-by  side.  Look  for  other  responses  that  are 
related  to  each  other  or  to  the  original  two  responses  that  were  set 
aside.  Repeat  this  process  until  you  have  all  of  the  responses 
placed  into  6-10  groupings.  Do  not  force-fit  single  responses  into 
groupings  in  which  they  don’t  belong.  These  “loners”  may  form 
their  own  grouping  or  never  find  a  home. 

Keep  the  following  in  mind: 

•  This  is  a  silent  process.  It  is  most  effective  to  have  everyone 
move  the  pieces  of  paper  at  will,  without  talking.  Talking 
sometimes  encourages  unhelpful  semantic  battles  that  can  rage 
on  and  on. 


Continued  on  next  page 
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•  Disagreements  over  the  placement  of  responses  should  be 
handled  simply  and  undiplomatically:  if  you  don’t  like  where  a 
response  is,  move  it!  It  will  all  eventually  settle  into 
consensus. 

Create  the  Category  Name  for  Each  Group 

5.  Look  for  a  response  in  each  grouping  that  captures  the  central 
idea  that  ties  all  of  the  responses  in  that  group  together.  This  is 
referred  to  as  a  header  card.  A  header  card  is  placed  at  the  top  of 
each  grouping.  Many  times,  no  such  card  exists,  and  a  header 
card  must  be  created.  The  header  card  should  be  concise. 

Construct  Frequency  Distribution  Chart  or  Table  of  Results 

6.  Develop  an  appropriate  representation  of  the  results  that  makes  it 
easy  to  compare  frequency  counts  within  each  category. 

Making 
comparisons 

For  example,  if  a  number  of  question  addressed  a  particular  topic  (e.g., 

requirements  management),  you  would  want  to  compare  responses 

from  question  to  question  to  determine  if  there  was  consistency. 


Once  all  data  has  been  compiled  and  transformed  into  something  easy 
to  interpret  (e.g.,  graphs,  charts,  cross-tabulations)  you  can  begin  to 
make  comparisons  between  responses  to  various  questions. 


Reviewing  related 
literature 


Based  on  the  type  of  survey  you  are  conducting,  you  might  want  to 
review  the  literature  so  that  you  can  compare  your  results  to  those  of 
other. 
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7.2 


Write  the  Report 


Introduction  By  now,  your  research  is  complete.  You  have  gathered  all  the  data 

together  and  have  developed  charts,  graphs,  and  tables  that  allowed 
you  to  make  observations  and  form  interpretations.  Now,  it  is  time  to 
begin  thinking  about  the  content  of  your  survey  report. 

The  survey  report  is  the  concrete  evidence  of  your  research  project.  In 
all  likelihood,  the  quality  of  your  research  will  be  judged  by  how  you 
write  this  report  that  conveys  your  ideas  and  the  findings  of  the  survey. 
For  this  reason,  it  is  important  to  carefully  consider  the  content, 
organization,  and  clarity  of  your  survey  report. 

Your  report  will  probably  take  a  number  of  different  forms,  depending 
on  the  particular  audience  you  are  addressing.  A  narrative  document 
(including  graphical  depictions  of  results)  will  be  your  key  information 
product.  But,  it  is  also  likely  that  you  will  develop  presentations  that 
address  various  aspects  of  your  research  work.  If  your  work  is  of 
interest  to  the  general  community,  you  might  also  want  to  write  journal 
articles  to  report  your  research  findings. 


Adjusting 
from  detail  to 
big-picture 


Up  until  now,  you  have  been  mired  in  minute  details  for  weeks  and 
perhaps  months — with  aspects  of  the  sampling  plan,  wording  choices 
for  questions,  appropriate  response  formats  for  questions,  testing  your 
implementations  in  various  ways,  and  finally  wrestling  with  the 
analysis  and  interpretation  of  the  results.  But  now,  you  need  to  take  the 
big-picture  view  and  let  go  of  a  lot  of  the  minutiae  that  has  been 
consuming  you.  The  task  at  hand  is  to  translate  your  research  work 
into  a  perspective  that  will  be  understood  by  your  audience. 
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7.2  Write  the  Report  (continued) 


Getting  started 


Conduct  an 
audience  analysis 


Before  diving  into  writing  your  report,  you  should  think  through  the 
information  that  you  have  to  present.  Sit  down  at  your  word  processor 
and  write  responses  to  the  questions  and  statements  listed  below.  The 
answers  you  provide  will  become  the  basis  for  your  report. 

•  What  was  the  purpose  of  this  survey  research? 

•  Make  a  list  of  people  who  contributed  to  the  research.  Could  they 
be  involved  in  writing  or  making  a  contribution  (as  a  reviewer)  to 
the  report?  What  contribution  could  they  make?  Plan  to 
acknowledge  contributors  and  the  sponsor  within  your  report  if 
appropriate. 

•  Compile  a  list  of  the  work  of  others  that  is  related  to  the  survey 
research  you  have  conducted. 

•  Describe  the  background  leading  up  to  the  research. 

•  What  factors  did  you  consider  to  be  threats  to  measurement 
validation? 

•  What  did  you  do  to  counter  known  threats  to  validation? 

•  Were  there  any  unanticipated  events  or  issues  that  arose  that  would 
threaten  the  validity  of  your  results? 

•  Describe  or  list  the  key  finding  of  your  research.  What  specific 
observations  (within  the  results)  led  you  to  these  findings? 

•  Describe  the  consequences  of  your  research.  How  will  the  results 
be  used?  Will  anyone  be  impacted  by  the  results?  If  so,  who? 

•  Do  you  have  recommendations  to  make  based  on  your  results? 
What  are  they? 

Answering  these  questions  will  help  you  get  started  writing  a  report. 
But  before  you  actually  begin  writing  it,  take  a  careful  look  at  the 
audience  that  you  believe  will  be  reading  it.  Who  are  they  and  what 
will  they  expect? 


It  is  very  important  to  take  stock  of  who  will  read  your  report — before 
you  begin  the  report  and  while  you  are  writing  it.  You  want  your  report 
to  communicate  well  to  your  audience,  so  you  need  to  consider  their 
information  needs.  Not  only  do  you  want  to  use  language  that  your 
audience  will  understand,  you  want  to  ensure  that  you  provide  the  right 
amount  of  detail.  In  some  cases,  you  might  have  more  than  one 
audience.  You  might  want  to  hide  detail  for  one  audience,  but  make  it 
accessible  by  providing  the  detail  in  an  appendix  for  your  other  audience. 
The  key  point:  write  for  the  readers  of  the  report — not  for  yourself. 

You  should  ensure  that  the  report  provides  enough  detail  about  your 
method  that  someone  skilled  in  survey  research  could  reproduce  your 
work.  But  it  is  also  best  to  assume  that  readers  are  busy  people  and 
they  will  want  information  presented  to  them  clearly  and  concisely. 

Thinking  about  your  audiences  before  you  write  the  report  can  help 
you  know  the  appropriate  level  of  detail  to  include  and  how  the  report 
should  be  organized. 
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7.2  Write  the  Report  (continued) 


Audience 
analysis: 
questions  to  ask 


Report 

organization 


You  might  want  to  ask  the  following  questions  while  conducting  your 

audience  analysis: 

•  Who  is  the  primary  audience?  What  positions  do  they  hold  in  their 
organizations  (e.g.,  executive,  management,  engineers)? 

•  Are  there  secondary  audiences?  If  so,  what  positions  do  they  hold 
in  their  organizations? 

•  Why  will  the  audience  want  to  read  the  report? 

•  What  does  the  audience  already  know  about  the  topic  that  your 
research  covers? 

•  What  is  the  most  important  thing  for  this  audience  to  understand 
from  reading  your  report? 

•  Are  there  important  terms  or  procedures  in  your  research  that  your 
audience  will  not  be  familiar  with?  List  them. 

After  you  draft  your  report,  return  to  your  audience  analysis  and  see  if 

you  have  addressed  the  points  you  raised  by  answering  the  questions 

above. 


Many  books  have  been  published  about  writing  research  reports.  Most 
research  reports  contain  the  same  major  sections,  although  the  names 
of  the  sections  vary  widely  and  it  is  appropriate  to  omit  some  sections 
and  add  others.  (The  audience  analysis  can  help  you  with  such 
decisions.) 

One  way  to  organize  your  report  is  as  follows: 

•  Title  page 

•  Table  of  Contents 

•  Abstract  (or  Executive  Summary) 

•  Introduction 

•  Research  Method 

•  Results 

•  Conclusions  and  Recommendations 

•  Reference  Section 

•  Appendix 

These  sections  are  described  in  upcoming  paragraphs. 
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7.2  Write  the  Report  (continued) 


Title  page 


Comment 

about 

authorship 


Table  of 
Contents 


As  you  begin  your  report,  you  may  start  with  a  working  title,  but  you 
should  be  open  to  changing  the  title  once  the  report  is  finished  so  that 
it  accurately  describes  your  research  paper. 

As  with  the  body  of  the  report  itself,  it  is  important  to  consider  your 
audience  when  developing  a  title  for  your  survey  report. 

Develop  a  title  that  will  make  your  audience  want  to  read  the  rest  of 
the  report.  Also  consider  whether  your  report  will  be  published  or 
available  on  the  Web  and  consider  including  a  keyword  in  the  title  to 
facilitate  successful  searches  for  your  document. 

As  part  of  the  title  page,  you  might  also  want  to  include  the  name  of 
the  organization  sponsoring  your  work,  the  date,  and  your  name. 
Depending  on  the  contribution  level  of  other  people,  you  might  want 
to  include  their  names  with  yours,  or  put  them  in  an 
“Acknowledgements”  section  of  your  paper. 


Tufte  spends  fifteen  pages  in  his  book.  Visual  Explanations,  talking 
about  the  Challenger  disaster  in  the  mid-1980s  [Tufte  97].  In  short,  the 
scientists  at  NASA  had  an  opportunity  to  stop  the  launch  of  the  ill- 
fated  space  shuttle.  Tufte  surmises  that  if  they  had  presented  the  data 
better,  the  scientists  might  have  been  able  to  convince  the  decision 
makers  not  to  push  the  button.  Although  he  spends  considerable  time 
in  the  details  of  the  actual  presentation,  Tufte  first  notes  that  the  title 
slide  did  not  include  the  names  of  the  authors. 

In  the  past,  many  of  us  have  noticed  that  “team  culture”  seems  to 
discourage  individual  ownership.  However,  this  habit  must  be  changed 
because,  as  Tufte  states  about  the  slides  in  the  challenger  presentation, 
“authorship  indicates  responsibility,  both  to  the  immediate  audience 
and  for  the  long-term  record.”  Without  any  accountability,  a  research 
report  “might  well  provoke  some  doubts  about  the  evidence  to  come.” 


The  table  of  contents  serves  two  purposes:  it  tells  your  readers  how  to 
navigate  the  document  and  it  demonstrates  how  the  report  is 
organized. 


CMU/SEI-2005-HB-004 


101 


7.2  Write  the  Report  (continued) 


Abstract 


Introduction 


An  abstract  is  an  accurate  representation  of  the  contents  of  a 
document,  in  an  abbreviated  form.  If  your  report  is  for  senior 
management,  then  your  abstract  becomes  an  “executive  summary.” 

An  abstract  can  be  difficult  to  write  because  you  must  describe  your 
subject  matter,  explain  what  was  done,  and  present  key  results  and 
interpretations,  all  in  one  short  paragraph  of  about  150  words. 

Although  the  abstract  appears  at  the  beginning  of  your  document,  it 
should  be  written  last  so  you  can  ensure  that  it  reflects  your  written 
report  accurately. 


The  Introduction  prepares  your  reader  for  the  report  that  follows 
by  presenting  the: 

•  research  purpose 

•  problem  statement 

•  scope 

•  background 

•  statement  of  the  hypothesis  or  hypotheses  if  applicable 

It  is,  of  course,  important  to  consider  the  purpose  of  your  research 
in  the  introduction. 

State  the  general  problem  area  and  discuss  the  importance  and 
significance  of  the  problem  area. 

The  scope  refers  to  the  ground  covered  and  not  covered  in  your  survey 
research.  It  can  also  help  the  reader  understand  limiting  factors  that 
constrained  your  research. 

The  background  includes  facts  that  the  reader  must  know  in  order  to 
understand  the  discussion  that  follows.  You  may  want  to  include 
elements  of  any  literature  review  that  you  conducted  in  the 
background. 

If  your  survey  was  designed  to  test  a  hypothesis,  then  clearly  state  that 
hypothesis  and  what  the  research  team  predicted. 
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7.2  Write  the  Report  (continued) 


Research 

Method 


Results 

section 


The  Research  Method  section  should  include  the  following 
subsections: 

•  Sampling 

•  Measurement 

•  Design  and  Procedures 

The  “Sampling”  subsection  should  clearly  describe  the  procedure  used 
to  select  the  respondents  in  the  survey.  The  population  (sampling 
frame)  should  be  identified  clearly.  The  specific  sampling  method 
should  be  identified  (e.g.,  self-selected  vs.  random)  and  the  sample 
itself  should  be  accurately  described  in  terms  of  number  and  any  other 
attributes  of  interest. 

The  “Measurement”  subsection  should  briefly  describe  the 
questionnaire  instrument  that  was  used.  The  instrument  itself  should 
appear  in  the  appendix.  Always  report  the  response  rate.  Discuss  any 
possible  threats  to  construction  validity  and  describe  what  you  did  to 
counteract  those  threats.  (Construct  validity  is  described  on  page  12.) 
Trochim  provides  an  excellent  detailed  discussion  of  construct  validity 
and  other  types  of  measurement  validity  issues  [Trochim  01]. 

The  “Design  and  Procedures”  subsection  should  describe  how  the 
research  was  conducted.  The  collection  method  should  be  described  in 
detail  (including,  for  example,  whether  incentives  were  used  to 
increase  the  response  rate).  The  entire  sequence  of  events  should  be 
described  such  that  another  skilled  researcher  could  replicate  the 
method.  Also,  use  this  subsection  to  discuss  threats  to  external  validity 
(see  page  12)  and  the  ways  that  you  either  mitigated  these  or  were  not 
able  to  control  for  them. 


In  the  Results  section,  you  will  finally  be  able  to  show  what  you 
discovered  or  observed  through  your  survey  research.  All  preceding 
sections  of  the  report  led  to  the  results  section  of  the  report  and  all 
subsequent  sections  will  refer  back  to  it  and  consider  what  the  results 
mean. 

In  the  research  section,  focus  on  the  facts  of  your  research  by 
presenting  them  in  a  straightforward  way.  You  will  need  to  decide 
whether  all  of  your  charts,  graphs,  and  figures  should  be  presented  in 
this  section  or  only  the  important  ones.  In  the  latter  case,  you  could 
place  some  information  in  an  appendix  if  it  seems  more  appropriate  for 
your  audience.  In  any  case,  you  should  provide  access  to  all 
information  collected  during  the  survey. 

Present  only  the  data  in  the  Results  section.  You  may  share  certain 
observations  here,  but  postpone  interpretations  and  suppositions  until 
the  next  section,  “Conclusions  and  Recommendations.” 


Continued  on  next  page 
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7.2  Write  the  Report  (continued) 


Results 

section 

(continued) 


Throughout  the  section  described  in  stage  7.1,  Analyze  the  Results,  the 
importance  of  charts,  graphs,  and  table  design  was  emphasized.  The 
design  of  your  presentations  is  all  the  more  important  when  you  are 
presenting  them  within  your  report.  Graphics  and  illustrations  need  to 
be  self-contained.  They  should  make  complete  sense  on  their  own 
without  reference  to  the  text.  After  drafting  your  Results  section, 
return  to  the  guidance  of  section  7.1  and  evaluate  the  quality  of  your 
illustrations  based  on  criteria  given. 

“Good  design  is  clear  thinking  made  visible,  bad  design  is  stupidity 
made  visible.”12 

-E.R.  Tufte 


Conclusions  &  This  section  is  sometimes  titled  Discussion,  but  the  more  descriptive 
Recommendations  title  Conclusions  and  Recommendations  better  informs  the  readers  of 
section  what  this  section  is  about.  In  this  section,  you  will  interpret  your 

results  as  a  whole.  If  appropriate  for  your  particular  research  topic, 
you  will  also  discuss  the  relationship  of  your  findings  to  earlier 
research.  Finally,  you  will  make  recommendations  based  on  the 
outcome  of  your  research. 

This  section  is  important  because  it  demonstrates  the  meaning  of  your 
research.  Your  interpretation  will  be  most  convincing  if  it  proceeds 
logically.  An  effective  way  of  doing  this  is  to  organize  your 
information  from  what  you  are  most  certain  about  to  what  you  are 
least  certain  about.  Consider  developing  your  argument  using  this 
outline: 

•  begin  with  a  discussion  of  the  data 

•  generalize  about  or  analyze  the  data 

•  consider  how  the  data  addresses  the  research  problem  or 
hypothesis 

•  discuss  what  can  be  inferred  from  the  data  as  they  relate  to  other 
research  (if  appropriate) 

•  make  recommendations  about  the  type  of  research  that  should 
follow  from  your  investigation  or  how  to  use  your  results 

An  important  aspect  of  your  report  is  the  identification  of  any 
limitations  of  your  research  and  their  nature.  This  is  especially 
important  when  your  results  are  inadequate  (e.g.,  due  to  poor  response 
rate),  do  not  support  your  hypothesis,  or  are  not  consistent  with  earlier 
studies  on  the  topic  that  you  researched.  Do  not  try  to  defend  your 
research  by  minimizing  the  seriousness  of  the  limitations,  but  instead 
focus  on  the  limitation  itself  as  it  affects  the  research  and  objectively 
try  to  account  for  it. 

Continued  on  next  page 


12  Tufte,  E.R.  “Presenting  Data  and  Information,”  [Workshop].  Arlington,  VA,  Nov.  9,  1999. 
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7.2  Write  the  Report  (continued) 


Conclusions  & 
Recommendations 
section 
(continued) 


The  References 
section  & 
Appendix 


Drawing  conclusions  follows  logically  from  your  interpretations  of  the 
data.  The  most  important  thing  is  to  state  your  conclusions  clearly.  Do 
not  be  ambiguous.  Be  careful  to  distinguish  between  conclusions 
(‘The  research  demonstrates  ...”)  and  inferences  (‘The  research 
suggests  ...”). 

Once  you  have  stated  your  conclusions,  discuss  their  implications. 
However,  your  research  may  have  actually  raised  more  questions  than 
it  answered.  Discussing  ambiguous  data  does  not  mean  that  your  work 
is  incomplete  or  faulty.  Rather,  it  demonstrates  your  non-bias  and  the 
intellectual  honesty  of  your  work,  connecting  it  to  the  larger  body  of 
scientific  methodology. 

Finally,  recommendations  should  be  listed  when  further  work  needs  to 
be  done.  In  many  situations,  a  single  survey  is  the  starting  point  for 
future  dialogue  and  studies  of  a  topic.  That  is  why  our  survey  process 
diagram  depiction  ends  with  an  arrow.  Your  recommendations 
demonstrate  that  you  fully  understand  the  importance  and  implications 
of  your  research  as  you  suggest  ways  that  it  can  be  continued. 


If  you  include  a  section  on  the  background  of  your  research  topic  or 
discuss  other  theories  or  models  related  to  your  work,  you  need  to 
include  a  References  section.  You  should  include  references  you  cited 
in  your  report  or  that  informed  your  research. 

The  appendix  is  a  place  for  information  that  is  relevant  to  your 
research  but  needs  to  be  kept  separate  from  the  main  body  of  the  report 
to  avoid  interrupting  the  line  of  development  of  the  report.  An 
appendix  should  contain  only  one  type  of  information.  If  you  plan  to 
include  several  sets  of  data,  then  include  several  appendices.  Label 
each  with  a  capital  letter  (e.g.,  Appendix  A,  Appendix  B,  Appendix  C). 
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7.2  Write  the  Report  (continued) 
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After  the  Survey 


introduction 


Is  your  work 
finished? 


Sharing  data 


You  have  completed  the  survey  and  written  your  report.  If  it  was  your 
first  experience  developing  a  survey,  you  are  probably  very  surprised 
that  it  was  so  much  work.  As  shown  by  List,  those  new  to  survey 
development  typically  underestimate  the  amount  of  effort  required  to 
design  and  develop  a  survey  and  analyze  the  results.  “While  it  is  easy 
to  write  a  questionnaire,  it  takes  skill  and  experience  to  write  a  good 
questionnaire:  one  in  which  every  question  is  clear,  can  be  answered 
accurately,  and  has  usable  results”  [List  05]. 


In  many  cases,  your  work  is  not  finished  with  the  release  of  your 
survey  report.  “Often  ‘truth’  will  not  win 
without  an  advocate”  [Backstom  81].  In 
almost  all  cases,  you  will  be  asked  to  deliver  a 
presentation  of  your  findings  to  management 
or  others.  Your  report  will  need  to  be 
repackaged  to  accommodate  this  type  of 
delivery. 

Be  aware  that  research  findings  can 
sometimes  contradict  the  pet  ideas  of 
colleagues  and  managers,  so  you  may  need  to  defend  the  research 
against  attacks  by  others  whose  ideas  are  threatened  by  the  research 
outcomes. 


Researchers  are  seldom  the  final  decision  makers.  But,  as  researchers, 
we  should  make  certain  that  the  survey  findings  are  given  a  hearing 
and  are  perceived  correctly. 


You  should  always  be  willing  to  share  information  about  your 
research.  However,  be  wary  of  how  certain  points  might  be  skimmed 
off  of  the  cream  of  your  careful  analysis  for  topical  reports  to  support 
others’  favored  points.  The  problem  is  not  that  your  data  is  being  used, 
but  that  it  is  being  used  without  providing  the  context  of  the  deeper 
analysis  and  caveats  that  you  may  have  cautioned  about  in  your  report. 

If  this  occurs,  all  you  can  do  is  address  the  issues  when  you  learn  of 
them  by  helping  the  user  of  your  data  understand  the  key 
underpinnings  of  your  research  that  will  assist  proper  interpretation. 
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After  the  Survey  (continued) 


Getting  to  the 
truth? 


Getting  to  the  truth  about  a  system  or  a  complex  situation  is  difficult. 

Data  triangulation  refers  to  the  use  of  multiple  data  collection  and 
analysis  methods  whereby  the  strengths  of  one  method  compensate  for 
the  weaknesses  of  another. 

The  purpose  of  triangulation  is  to  obtain  confirmation  of  findings 
through  convergence  of  different  perspectives.  The  point  at  which  the 
perspectives  converge  is  seen  to  represent  reality. 

As  time  goes  on,  your  survey  will  be  looked  upon  to  add  information 
to  an  existing  body  of  knowledge  that  builds  upon  many  sources  of 
information  to  establish  the  truth. 

Surveys  must  be  constructed  properly  in  order  for  them  to  withstand 
the  test  of  time. 
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Appendix  A:  Approaches  to  Data  Gathering 


Introduction 


Approaches  to 
data  gathering 


Documentary 

research 


Laboratory 

experiments 


The  data-gathering  approach  chosen  should  be  dictated  by  the 
information  needs  of  the  researcher.  Different  approaches  produce 
different  kinds  of  information. 

This  appendix  describes  various  research-based  approaches  for 
gathering  data. 


Survey  research  is  but  one  approach  for  collecting  information  to  gain 
insight  into  the  people  and  problems  under  study.  Other  approaches 
include  the  following: 

•  documentary  research 

•  laboratory  experiments 

•  action  research 

•  case  studies 

•  field  experiments 

•  field  work:  participant  observation  and  ethnography 

•  simulation 

•  in-depth  interviews 

The  best  approach  to  use  depends  on  the  information  needed  for  the 
problem  at  hand. 


One  of  the  first  things  researchers  should  determine  is  whether  the 
information  they  are  seeking  already  exists.  Checking  sources  such  as 
the  Web,  libraries,  public  offices,  and  private  associations  might  help 
researchers  avoid  conducting  redundant  and  possibly  wasteful  surveys 


A  laboratory  experiment  with  human  subjects  can  be  designed  to 
explain  behavior  by  noting  what  happens  as  some  new  element  is 
introduced.  Subjects  used  are  typically  those  who  signs  up  voluntarily. 
Conclusions  drawn  from  such  experiments,  therefore,  cannot  be 
applied  to  just  anyone,  anywhere  [Backstrom  81]. 
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Appendix  A:  Approaches  to  Data  Gathering  (continued) 


Action  research 


Case  studies 


Field 

experiments 


Field 

work 


Simulation 


Action  research  involves  a  family  of  research  methodologies  that 
pursue  action  (i.e.,  change)  and  research  (i.e.,  understanding)  at  the 
same  time  [Dick  99].  In  most  of  its  forms,  action  research  involves 

•  using  a  cyclic  or  spiral  process  which  alternates  between  action 
and  critical  reflection 

•  continuously  refining  methods,  data,  and  interpretation  in 
subsequent  cycles,  in  the  light  of  the  understanding  developed  in 
the  earlier  cycles 


A  case  study  is  an  intensive  study  of  a  specific  individual  or  context 
[Trochim  01].  There  is  no  single  way  to  conduct  a  case  study.  A 
combination  of  methods  (such  as  unstructured  interviewing  and  direct 
observation)  can  be  used.  Friedman  provides  a  fine  example  of  case 
study  use  in  the  domain  of  systems  engineering  and  management  in 
acquisition  systems  [Friedman  04]. 


When  conducting  field  research,  the  researcher  goes  into  the  field  to 
observe  the  phenomenon  in  its  natural  state.  The  field  researcher 
typically  takes  extensive  field  notes  that  are  subsequently  coded  and 
analyzed  [Trochim  01].  Controlled  field  experiments  require  a  special 
kind  of  rigorous  structure,  called  experimental  design.  Although  the 
experiments  occur  in  the  field,  their  form  is  like  that  of  a  chemistry 
experiment  in  the  laboratory.  The  researcher  observes  a  real-world 
group  (the  “before”),  adds  a  new  ingredient  (the  “treatment”)  and 
observes  any  changes  (the  “after”)  [Backstrom  81]. 


The  ethnographic  approach  to  qualitative  research  comes  largely  from 
the  field  of  anthropology.  The  researcher  becomes  immersed  in  the 
culture  as  an  active  participant  and  records  extensive  field  notes.  In  an 
ethnographic  study  there  is  no  preset  limiting  of  what  will  be  observed 
and  no  predefined  ending  point  [Trochim  01], 


Simulation  is  usually  a  mathematical,  computer-generated  model  of 
the  real  world.  By  duplicating  the  “world”  with  statistical  techniques 
and  by  varying  the  values  of  the  model  parameters,  this  approach 
attempts  to  predict  how  conditions  will  change  in  response  to  new 
values.  Simulations  can  be  generalized  to  the  extent  that  the  data  and 
relationships  used  in  them  have  been  previously  verified  by  separate 
research  [Backstrom  81]. 
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Appendix  A:  Approaches  to  Data  Gathering  (continued) 


In-depth 

interviews 


The  in-depth  interview  is  an  exploratory  approach.  Each  topic  is  probed 
deeply — following  up  important  leads  with  further  questions — until  a 
well-rounded  picture  emerges.  The  results  of  this  time-consuming  inquiry 
cannot  be  applied  to  a  larger  group  because  one  cannot  know  how 
representative  the  respondents  are  of  the  larger  group  [Backstrom  81]. 


Comparison  of 
approach  by 
information  goals 


The  figure  below  summarizes  a  comparison  of  major  research 
approaches  by  information  goals  (adapted  from  Backstrom  81, 
page  11). 


No  generalization 


Generalization 


Describes  what 
behavior  exists 

•  Depth  interview 

•  Action  research 

•  Field  work 

•  Survey 

•  Case  study 

•  Controlled  field 

Describes  what 

•  Human  laboratory 

experiment 

causes  behavior 

experiment 

•  Simulation 

•  Physical  laboratory 

experiment 
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Appendix  A:  Approaches  to  Data  Gathering  (continued) 


114 


CMU/SEI-2005-HB-004 


Appendix 


Introduction 


Problem 

identification 


B:  Applications  of  Survey 
Research  in  Systems  and 
Software  Engineering 


Assumptions  are  your  personal  interpretations  of  phenomena  or 
information  that  you  have  about  a  system.  When  this  information  is 
based  on  casual  study,  hearsay,  untested  inferences,  or  limited 
experience,  the  assumptions  on  which  you  act  may  not  be  sound.  That 
is  why  structured  and  systematic  data-collection  and  analysis 
approaches  must  be  used,  such  as  those  available  through  survey 
research. 

The  use  of  properly  designed  surveys  can  help  provide  needed  insight 
into  complex  systems  and  software  engineering  issues  in  a  timely  and 
accurate  way  to  support  more  effective  problem-solving  and  decision¬ 
making. 

The  purpose  of  this  appendix  is  to  provide  some  examples  of  common 
ways  surveys  have  been  used  in  software  and  systems  engineering 
environments. 


In  order  to  stay  relevant  and  competitive  in  the  global  economy, 
organizations  must  continually  seek  ways  to  improve  their  people, 
processes,  and  technologies. 

But  what  are  the  high-leverage  improvement  opportunities  that  can 
provide  the  greatest  impact?  To  answer  this  question  (at  least  in  part), 
surveys  can  gather  the  multi-level  perspectives  of  the  organization. 

For  example: 

Under  the  auspices  of  the  Under  Secretary  of  the  Army,  a  large-scale 
program  was  initiated  for  the  purpose  of  making  dynamic 
improvements  in  the  Army’s  acquisition  system. 

While  there  are  many  changes  that  could  be  made  to  the  system,  there 
were  many  different  opinions  as  to  what  the  focus  should  be  for 
improvement  during  the  next  several  years. 

Therefore,  a  survey  was  designed  and  distributed  to  nearly  all  Army 
Acquisition  program  managers  to  elicit  what  they  believed  were  the 
key  problems  and  risks  they  faced  each  day  [Kasunic  04]. 
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Appendix  B:  Applications  of  Survey  Research  in  Systems  and  Software 
Engineering  (continued) 


Employee 
and  customer 
satisfaction 


It  is  in  the  best  interests  of  companies  and  organizations  to  maintain 
optimal  relationships  with  both  their  employees  and  customers.  Many 
organizations  conduct  periodic  surveys  to  understand  what  is  working 
well  and  what  needs  to  improve. 


Process 

improvement 

validation 


Many  organizations  are  working  to  improve  the  way  they  do  business 
and  are  engaged  in  various  initiatives  to  do  so  (e.g.,  CMMI,  Six 
Sigma,  software  architecture  design  improvement,  interoperability). 

While  the  best  avenue  to  assessing  the  success  of  improvement  efforts 
lies  in  first-hand  performance  measurement  of  key  characteristics  of 
interest,  a  more  rounded-out  picture  will  emerge  if  surveys  are  used  to 
gauge  the  perceptions  of  those  affected  by  the  intended  improvement 
change. 

For  example,  consider  a  process  that  is  to  be  improved.  To  better 
understand  whether  the  change  intervention  accomplished  its 
objective,  a  survey  questionnaire  could  be  administered  before  and 
after  the  change.  A  comparison  of  the  results  of  the  before  and  after 
surveys  can  help  assess  the  perceived  impact  of  the  change,  as 
illustrated  in  the  example  diagram  below. 


Magnitude  of  improvement 
h - M 


The  new  state  of  practice 
after  improvement  intervention 


Example  of  survey  results  before  and  after  improvement  change  is  made. 
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Appendix  B:  Applications  of  Survey  Research  in  Systems  and  Software 
Engineering  (continued) 


Understanding 
the  state  of 
the  practice  & 
community  needs 


Quite  often,  surveys  are  used  to  gauge  the  state  of  the  practice  across  the 
systems  and  software  engineering  community  or  a  portion  of  it.  The 
purpose  of  these  surveys  is  to  uncover  data  that  can  be  used  to  identify 
and  prioritize  the  needs  of  the  community.  In  this  way,  decision-making 
can  be  informed  so  that  research,  technology,  or  product  and  service 
development  can  be  channeled  in  those  areas  that  perceive  to  be  most 
relevant  to  the  stakeholders. 


The  following  documents  provide  examples  of  these  types  of  surveys: 

•  Pittsburgh  Regional  Delphi  Study  -  Software  and  Corporate 
Technology  [Tide  04]. 

•  The  1999  Survey  of  High  Maturity  Organizations  [Paulk  99]. 

•  After  the  Appraisal:  A  Systematic  Survey  of  Process  Improvement, 
its  Benefits,  and  Factors  that  Influence  Success  [Goldenson  95]. 


Short  topical 
surveys  with 
limited 
distribution 


Small  group  surveys  are  often  administered  to  obtain  feedback  and 
improvement  suggestions.  Here  are  some  examples: 

•  evaluation  questionnaire  after  attending  a  training  course 

•  meeting  evaluation  questionnaire 

•  questionnaire  to  obtain  feedback  after  a  new  innovation  has  been 
introduced  into  your  organization 

•  questionnaires  about  particular  topics  of  interest  in  the  workplace 

These  types  of  surveys  are  usually  given  to  everyone  within  a  group.  A 
survey  that  includes  the  entire  population  in  the  sample  is  called  a 
“census.” 

Even  though  a  questionnaire  might  be  targeted  to  a  small  group  of 
people,  (e.g.,  a  class  of  25  people  who  are  evaluating  a  training 
course),  the  researcher  should  take  care  in  designing  the  instrument  so 
that  the  information  obtained  is  meaningful  and  interpretable. 
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Appendix  B:  Applications  of  Survey  Research  in  Systems  and  Software 
Engineering  (continued) 
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Appendix  C:  Example  Activity  Breakdown 

for  a  Survey  Research  Project 


The  example  activity  breakdown  structure  below  is  for  organizing  survey  research  work.  The 
list  of  activities  and  responsibilities  need  to  be  tailored  based  on  the  type  of  survey  intended 


Activity 

Responsibility 

Identify  &  characterize  research  problem 

Define  and  document  research  objectives 

Researcher,  SMEs13 

Define  and  document  research  questions 

Researcher,  SMEs 

Conduct  review  with  management 

Researcher,  SMEs,  Management 

Define  and  document  how  survey  results  will  be  used 

Researcher,  SMEs,  Management 

Map  research  questions  to  objectives 

Researcher 

Review  and  finalize  objectives  and  internal  questions 

Researcher,  SMEs,  Management 

Is  a  survey  research  approach  appropriate?  If  so  continue 
with  plan. 

Define  population  (target  audience) 

Define  appropriate  audience  for  survey  (sampling  frame) 

Researcher,  SMEs 

Verify  that  sampling  frame  can  be  enumerated 

Researcher 

Obtain  contact  information  for  sampling  frame 

Researcher 

Conduct  audience  analysis  and  document 

Researcher,  SMEs  I 

Determine  appropriate  media  for  target  audience  (Web- 
based,  mail,  email,  etc.) 

Researcher,  SMEs 

Recruit  members  of  target  audience  to  participate  in  pilot 
test  of  survey14 

Researcher  j 

Design  sampling  plan 

Define  desired  confidence  and  precision 

Researcher 

Select  sample 

Researcher 

Document  sampling  approach 

Researcher 

Review  sampling  approach 

Researcher,  SMEs 

Sampling  plan  approved 

13  Subject  Matter  Experts  Continued  on  next  page 

14  Although  pilot  test  des  not  happen  until  much  later  in  the  development,  it  is  desirable  to  secure 
commitment  early  on. 
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Appendix  C:  Example  Activity  Breakdown  for  a  Survey  Research  Project 
(continued) 


Activity 

Responsibility 

Develop  paper-based  questionnaire 

Translate  research  questions  into  appropriate  questions  for 
respondents 

Researcher 

Select  question  type/format  and  specify  wording  for  each 
question 

Researcher 

Review  questions  and  prioritize  list  of  questions.  Pare  down 
list  to  fit  size  constraints. 

Researcher,  SMEs 

Organize  and  sequence  questions 

Researcher 

Write  instructions,  definitions,  and  transition  paragraphs 

Researcher 

Conduct  team  review 

Researcher,  SMEs,  Management 

Revise  questionnaire  based  on  review 

Researcher 

Conduct  external  review  with  other  stakeholders15 

Researcher,  other  stakeholders 

Revise  questionnaire  based  on  external  review 

Researcher 

Conduct  internal  pre-test  of  paper  version  of  questionnaire 

Researcher,  SMEs 

Revise  questionnaire  based  on  internal  pilot  test 

Researcher 

Final  edit  of  paper-based  questionnaire 

Technical  Writer,  Researcher 

Final  draft  of  paper-based  version  is  completed 

Develop  ancillary  survey  documents 

Write  pre-notification  letter 

Researcher 

Write  cover  letter 

Researcher 

Write  follow-up  reminder  notes 

Researcher 

Conduct  team  review 

Researcher,  SMEs,  Management 

Revise  based  on  team  review 

Researcher 

Conduct  final  edit  of  ancillary  survey  documents 

Technical  Writer,  Researcher 

Ancillary  documents  completed 

Continued  on  next  page 


15  It  is  a  good  idea  to  obtain  buy-in  from  other  subject  matter  experts  who  are  not  participating  on 
the  team.  Obtaining  buy-in  at  this  early  stage  will  increase  belief  or  acceptance  of  the  survey  re¬ 
sults. 
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Appendix  C:  Example  Activity  Breakdown  for  a  Survey  Research  Project 
(continued) 


Activity 

Responsibility 

Develop  requirements  for  Web-based  version  of 
questionnaire16 

Define  questionnaire  access  mechanism 

Researcher 

Define  interrupt/resume  requirements 

Researcher 

Define  response  format  requirements 

Researcher 

Specify  screen  layout  desires 

Researcher 

Specify  output  requirements 

Researcher 

Meet  with  survey  tool  programmer  to  discuss  requirements 
and  agree  on  implementation 

Researcher,  Web  Programmer 

Document  meeting  outcomes  and  agreements 

Researcher 

Develop  Web-based  version  of  questionnaire 

Write  content  for  introduction  screen 

Researcher 

Enter  all  information  into  web-based  survey  tool 

Web  Programmer 

Conduct  review 

Researcher,  SMEs 

Revise  web  questionnaire 

Researcher,  Web  Programmer 

Conduct  usability  test 

Researcher,  SMEs 

Revise  web  questionnaire  based  on  test 

Researcher,  Web  Programmer 

Conduct  Pilot  Test 

Develop  evaluation  form  for  pilot  test 

Researcher 

Conduct  pilot  test 

Researcher,  Pilot  Test  Participants 

Evaluate  feedback 

Researcher,  SMEs 

Revise  both  paper-based  version  of  questionnaire  and  Web- 
based  version 

Researcher,  Web  Programmer 

Conduct  final  edit 

Conduct  team  review 

Researcher,  SMEs,  Management 

Revise  based  on  team  review 

Researcher 

Final  edit  before  questionnaire  distribution 

Technical  Writer,  Researcher 

Questionnaire  ready  for  distribution 

_  Continued  on  next  page 

16  The  assumption  here  is  that  a  web-based  survey  application  is  pre-existent  and  has  been  used  al¬ 
ready  within  the  organization.  If  this  is  not  the  case,  then  the  research  team  would  need  to  conduct 
an  evaluation  of  available  technologies  and  select  the  appropriate  application  to  support  the  re¬ 
search  needs. 


CMU/SEI-2005-HB-004 


121 


Appendix  C:  Example  Activity  Breakdown  for  a  Survey  Research  Project 
(continued) 


Activity 

Responsibility 

Distribute  questionnaires  &  track  responses 

Distribute  questionnaires 

Researcher 

Track  responses 

Researcher 

Send  reminder  #1 

Researcher 

Send  reminder  #2  • 

Researcher 

Response  deadline 

Process  survey  responses 

Develop  graphical  depictions  of  response  information 

Researcher 

Conduct  analysis 

Researcher 

Write  report 

Write  initial  draft 

Researcher 

Conduct  team  review  of  draft 

Researcher,  SMEs 

Revise  draft 

Researcher 

Review  and  provide  mark-up  of  report 

Technical  Writer 

Revise  report 

Researcher 

Conduct  final  team  review 

Researcher,  SMEs 

Conduct  final  edit 

Technical  Writer,  Researcher 

Report  complete 
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Appendix  D:  Likert-Type  Response 

Alternatives 


Agreement 


•  Strongly  agree 

•  Agree 

•  Agree  very  strongly 

•  Agree 

•  Undecided 

•  Agree  strongly 

•  Undecided 

•  Disagree 

•  Agree 

•  Disagree 

' 

•  Disagree 

•  Strongly  disagree 

•  Disagree  strongly 

•  Disagree  very  strongly 

Frequency 


•  Very  frequently 

•  Always 

•  Almost  always 

•  Frequently 

•  Very  frequently 

•  To  a  considerable  degree 

•  Occasionally 

•  Occasionally 

•  Occasionally 

•  Rarely 

•  Rarely 

•  Seldom 

•  Very  Rarely 

•  Very  rarely 

•  Never 

•  Never 

• 

Often 

•  Always 

• 

A  Great  Deal 

• 

Sometimes 

•  Very  Often 

• 

Much 

• 

Seldom 

•  Sometimes 

• 

Somewhat 

• 

Never 

•  Rarely 

• 

Little 

•  Never 

• 

Never 

Continued  on  next  page 
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Appendix  D:  Likert-Type  Response  Alternatives  (continued) 


Quality 


•  Excellent 

•  Very  Good 

•  Good 

•  Above  Average 

•  Good 

•  Fair 

•  Average 

•  Barely  Acceptable 

•  Poor 

•  Below  Average 

•  Poor 

•  Extremely  Poor 

•  Very  Poor 

•  Very  Superior 

•  Very  adequate 

•  Totally  adequate 

•  Superior 

•  Slightly  adequate 

•  Very  adequate 

•  Borderline 

•  Borderline 

•  Barely  adequate 

•  Poor 

•  Slightly  inadequate 

•  Borderline 

•  Very  poor 

•  Very  inadequate 

•  Barely  inadequate 

•  Very  inadequate 

•  Totally  inadequate 

Likelihood 


•  Almost  Always  True 

•  Definitely 

•  To  A  Great  Extent 

•  Usually  True 

•  Very  Probably 

•  Somewhat 

•  Often  True 

•  Probably 

•  Very  Little 

•  Occasionally  True 

•  Possibly 

•  Not  At  All 

•  Sometimes  But 

•  Probably  Not 

Infrequently  True 

•  Very  Probably  Not 

•  Usually  Not  True 

•  Almost  Never  True 
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Appendix  E:  Example  Email  Cover  Letters 


The  letter  below  is  from  the  NQAA  Software  Survey  [NOAA  96]. 


NOAA  is  conducting  a  survey  to  gain  a  better  understanding  of  its  current  software  systems  and 
software  activities.  To  ensure  that  NOAA’s  software  management  process  is  an  integral  part  of 
NOAA’s  Information  Technology  planning  program,  the  Systems  Acquisition  Office  is  working  in 
collaboration  with  the  Information  Systems  and  Finance  Office  and  the  Environmental  Data  and 
Information  Services  Strategic  Planning  Team  to  conduct  this  software  survey.  A  Software  Survey 
Team  composed  of  representatives  from  NOAA  Line  and  Staff  Offices  have  worked  to  develop  and 
implement  this  survey. 

The  survey  results  are  not  intended  to  invoke  mandated  “improvement”  standards  and  processes.  The 
understanding  gained  from  the  survey  will  be  used,  first,  to  identify  “best  practices”  which  currently 
exist  within  NOAA,  second  to  define  otential  improvements,  and,  third,  to  serve  as  a  reference 
against  which  future  comparisons  are  made  as  candidate  improvements  are  prototyped  and 
implemented. 

If  you  have  any  questions  at  any  time  regarding  this  activity,  please  contact  XXXX  or  XXXX  of  the 
Systems  Acquisition  Office  at  (XXX)  XXX-XXXX. 

Your  responses  to  this  questionnaire  are  important  to  us.  They  will  help  us  develop  the  most 
complete  and  accurate  understanding  possible  of  software  within  NOAA.  The  questionnaire  should 
take  no  more  than  one  hour  to  complete.  When  you  are  finished  completing  the  questionnaire,  please 
promptly  return  it  to  XXX. 

Thank  you  for  taking  the  time  to  complete  this  questionnaire. 
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Appendix  E:  Example  Email  Cover  Letters  (continued) 


The  letter  below  is  an  adaptation  of  an  example  provided  by  Couper  [Couper  01]. 


From:  Michigan  Daily  Survey 

Sent:  Tuesday,  March  30,  1999  2:52  PM 

To:  [e-mail  address] 

Subject:  Michigan  Daily  Student  Survey 
Dear  University  of  Michigan  Student: 

You  have  been  selected  at  random  as  part  of  a  sample  of  1,600  currently  enrolled  students  at  The 
University  of  Michigan  to  participate  in  a  survey  organized  by  The  Michigan  Daily.  We  are 
interested  in  campus  life  and  in  your  opinions  about  some  University  of  Michigan  policies  that  affect 
students,  including  affirmative  action.  We  need  your  answers  to  make  our  survey  a  more  accurate 
representation  of  student  views.  Your  participation  in  the  study  is  completely  voluntary,  and  your 
answers  will  not  be  linked  to  you  by  name  in  anyway.  The  overall  results  from  the  survey  will  be 
presented  in  series  of  articles  in  The  Michigan  Daily. 

You  can  complete  the  survey  whenever  it  is  convenient  for  you  at  any  computer  where  you  can  get 
access  to  the  World  Wide  Web.  It  will  take  you  about  15  minutes.  The  information  you  provide  us  is 
very  important  to  the  accuracy  of  our  survey.  No  one  from  the  study  staff  will  be  able  to  connect  you  to 
your  answers.  As  a  token  of  our  appreciation,  we  would  like  to  offer  you  a  gift  to  thank  you  for  your 
participation  in  the  survey.  When  you  complete  the  survey,  you  will  receive  instructions  about  how  to 
claim  a  copy  of  The  Daily’s  color  book  We’re  No.  1:  The  1997  National  Championship  Season. 

Thank  you  in  advance  for  completing  the  survey.  You  are  contributing  to  the  discussion  of  important 
campus  issues  that  will  begin  when  the  survey  results  are  published  in  The  Daily. 

Please  use  MS  Explorer  or  Netscape  versions  4  or  higher. 

Enter  this  unique  CASE  IDENTIFICATION  NUMBER:  [ID] 

Use  this  PASSWORD  to  verify  your  eligibility  as  a  study  participant:  [password] 

To  start  the  survey,  just  go  to:  http://survey.isr.umich.edu/michsurvey/welcome.htm 

If  you  have  questions  about  the  survey  or  are  having  trouble  signing  on  to  the  web  site,  please 
contact  either  one  of  us. 

Michael  W.  Traugott 

Department  of  Communication  Studies 

xxxxx  @  umich.edu 

Jennifer  Yachnin 

The  Michigan  Daily 

xxxxx@umich.edu 
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